Tag: Spark

Feed: James Serra's Blog. Author: James Serra. With data lakes becoming popular, and Azure Data Lake Store (ADLS) Gen2 being used for many of them, a common question I am asked about is “How can I access data in ADLS Gen2 instead of a copy of the data in another product (i.e. Azure SQL Data Warehouse)?”. The benefits of accessing ADLS Gen2 directly is less ETL, less cost, to see if the data in the data lake has value before making it part of ETL, for a one-time report, for a data scientist who wants to use the data to ... Read More

Feed: Neo4j Graph Database Platform. Author: Jocelyn Hoppa. The votes are in.This past June, national standards bodies around the world – belonging to ISO/IEC’s Joint Technical Committee 1 (responsible for IT standards) – began voting on the GQL project proposal.Graph Query Language (GQL) is a new language being developed and maintained by the same international working group that also maintains the SQL standard.Now, it’s official: Earlier this week, the ballot closed and the proposal passed, with seven countries putting experts forward to work on the four-year project.GQL draws heavily from existing languages, but the main inspiration has so far been ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: ajit jaokar. I have been looking at this problem over a few years now The IoT industry often speaks of handling both high volumes and high throughputs of data However, currently, I find that there are not many use cases for IoT streaming analytics which are unique The 'unique' and 'currently' bits are important i.e. identifying use cases that need the analytics primarily implemented in the stream (and not applied to data when at rest) Do you know of companies/ vendors actually implementing this? (in production today) On one hard, every ... Read More

Feed: Planet big data. Author: Brandon Purcell. As a data and analytics leader, you have the means to help your company reimagine your business. There’s an opportunity for you to become a meaningful change agent in your organization. Not sure where or how to start? We’ll help you get there at this year’s Data Strategy & Insights Forum in Austin, Texas. Reimagine To Reinvent In most organizations, data strategy and insights professionals and their teams are seen as back-office functions, running numbers to validate and prove. They are known to be worker bees: working predominantly behind the scenes; heads down; curious and abreast of new research ... Read More

Feed: AWS Big Data Blog. Perimeter security helps secure Apache Hadoop cluster resources to users accessing from outside the cluster. It enables a single access point for all REST and HTTP interactions with Apache Hadoop clusters and simplifies client interaction with the cluster. For example, client applications must acquire Kerberos tickets using Kinit or SPNEGO before interacting with services on Kerberos enabled clusters. In this post, we walk through setup of Apache Knox to enable perimeter security for Amazon EMR clusters.It provides the following benefits: Simplify authentication of various Hadoop services and UIs Hide service-specific URLs/Ports by acting as a Proxy Enable SSL ... Read More

Feed: Planet big data. Author: Nigel Fenwick. Let’s face it, digital transformation efforts frequently over-promise and under-deliver. Many “transformations” fail to deliver on expectations simply because they are not really transformative. Upgrading digital experiences and creating new mobile apps delivers a level of differentiation that is short-lived.  At the speed of digital business, competitors quickly match or exceed your efforts. Many digital leaders and vendors enthusiastically label any digitization project as a digital transformation. But we’ve been digitizing business for 30+ years. Simply upgrading technology, creating new digital experiences, or moving legacy apps to the cloud are not digital business ... Read More

Feed: Neo4j Graph Database Platform. Author: Jocelyn Hoppa. Almost a decade ago, Neo4j took off as a transactional graph database management platform. Just a year after its initial launch, a new, powerful query language called Cypher was introduced to allow expressive, efficient querying and other interactions with the graph data in Neo4j. Today, Neo4j supports graph analytics including implementation of a wide variety of state-of-the-art graph algorithms with custom optimizations for Neo4j’s graph database model. It achieves unmatched performance in scalability, routinely running millions or even billions of nodes and edges.The next natural step for the platform in 2019 is ... Read More

Feed: Radar. Author: Mac Slocum. Attend the Strata Data Conference in New York, September 23-26 to learn how ML, AI, and data are converging. The Strata Data Conferences helped chronicle the birth of big data, as well as the emergence of data science, streaming, and machine learning (ML) as disruptive phenomena. Strata attracts the leading names in the fields of data management, data engineering, analytics, ML, and artificial intelligence (AI). For nearly a decade, it’s provided a venue for developers, data and ML engineers, data architects, data scientists, and others to acquire or hone skills, explore provocative ideas, and network ... Read More

Feed: Microsoft Azure Blog. Author: Tyler Fox. This is the fourth blog post in a four-part series on monitoring on Azure HDInsight. Monitoring on Azure HDInsight Part 1: An Overview discusses the three main monitoring categories: cluster health and availability, resource utilization and performance, and job status and logs. Part 2 is centered on the first topic, monitoring cluster health and availability. Part 3 discussed monitoring performance and resource utilization. This blog covers the third of those topics, workload metrics and logs, in more depth. During normal operations when your Azure HDInsight clusters are healthy and performing optimally, you will likely ... Read More

Feed: Recent Announcements. Today, PySpark and SparkR users must install their dependencies on each host in a cluster. As a result, teams operating multi-tenant clusters struggle to keep up with specific library versions and manage dependencies, limiting developer productivity, increasing the time spent preparing a cluster for use, and adding complexity to cluster upgrades.  Using Hadoop 3, Docker, and EMR, Spark users no longer have to install library dependencies on individual cluster hosts, and application dependencies can now be scoped to individual Spark applications. This is achieved by running Spark applications in Docker containers instead of directly on EMR cluster ... Read More