Category: Big Data

Feed: AWS Big Data Blog. Data is being produced, streamed, and consumed at an immense rate, and that rate is projected to grow exponentially in the future. In particular, JSON is the most widely used data format across streaming technologies and workloads. As applications, websites, and machines increasingly adopt data streaming technologies such as Apache Kafka and Amazon Kinesis Data Streams, which serve as a highly available transport layer that decouples the data producers from data consumers, it can become progressively more challenging for teams to coordinate and evolve JSON Schemas. Adding or removing a field or changing the data ... Read More

Feed: AWS Big Data Blog. Streaming ETL jobs in AWS Glue can consume data from streaming sources such as Amazon Kinesis and Apache Kafka, clean and transform those data streams in-flight, as well as continuously load the results into Amazon Simple Storage Service (Amazon S3) data lakes, data warehouses, or other data stores. The always-on nature of streaming jobs poses a unique challenge when handling fast-changing reference data that is used to enrich data streams within the AWS Glue streaming ETL job. AWS Glue processes real-time data from Amazon Kinesis Data Streams using micro-batches. The foreachbatch method used to process ... Read More

Feed: Cloudera Blog. Author: Arielle Diamond. Posted in Business | January 24, 2022 3 min read Many organizations initiate data projects because they want to increase revenue, but a select few tackle projects that truly transform society.  This year, Cloudera is recognizing three organizations as finalists in the Data for Good category of its annual Data Impact Awards: Union Bank of the Philippines, Keck Medicine of USC, and the National Bone Marrow Donor Program. In order to be considered for the Data for Good category, submissions must have addressed some of the most challenging issues affecting society and the planet, ... Read More

Feed: Actian. Author: Lewis Carr. And why is it better than a Data Lake or an Analytics Hub? In the opening installment of this blog series—Data Lakes, Data Warehouses and Data Hubs: Do we need another choice?  I explore why simply migrating these on-prem data integration, management, and analytics platforms to the Cloud does not fully address modern data analytics needs. In comparing these three platforms, it becomes clear that while all of them meet certain critical needs, none of them meet the needs of business end-users without significant support from IT. What we in fact need is a platform ... Read More

Feed: Actian. Author: Pradeep Bhanot. In this third and last part of the series on Actian Vector in Hadoop (VectorH), we will cover how the VectorH file format supports the performance and efficiency of our data analytics platform to accelerate business insights, as well as some of the other enterprise features that can help businesses move their Hadoop applications into production. Part one of this series showed the huge performance advantages VectorH has over other SQL on Hadoop alternatives, while part two explored the benefits of the richer implementation of SQL and the ability to perform data updates in VectorH.The ... Read More

Feed: Domo Blog. Author: Jace McLean.  / What 2 Surveys Say About the Value of Centralized Data Back in October 2021, we took a look at some data we collected at Big Data London (BDL) related to companies’ data strategies. Since then, we’ve collected even more data on the subject. At Web Summit in November, we set up the same survey machine we took to BDL, and again asked attendees a few questions about their data environments. You can find the updated dashboard (aka the Swag Machine) toward the bottom of this post. But first, let’s zoom in on the same ... Read More

Feed: Cloudera Blog. Author: Niel Dunnage. Posted in Technical | January 21, 2022 9 min read This blog will summarise the security architecture of a CDP Private Cloud Base cluster. The architecture reflects the four pillars of security engineering best practice, Perimeter, Data, Access and Visibility. The release of CDP Private Cloud Base has seen a number of significant enhancements to the security architecture including: Apache Ranger for security policy management Updated Ranger Key Management service Before diving into the technologies it is worth becoming familiar with the key security principle of a layered approach that facilitates defense in depth ... Read More

Feed: Alation. Author: Myles Suer. January 20, 2022 — Several weeks ago (prior to the Omicron wave), I got to attend my first conference in roughly two years: Dataversity’s Data Quality and Information Quality Conference. Ryan Doupe, Chief Data Officer of American Fidelity, held a thought-provoking session that resonated with me. In Ryan’s “9-Step Process for Better Data Quality” he discussed the processes for generating data that business leaders consider trustworthy. To be clear, data quality is one of several types of data governance as defined by Gartner and the Data Governance Institute. Quality policies for data and analytics set ... Read More

Feed: Sand Hill Group. Author: clare. Elay Cohen is the Co-Founder and CEO of Saleshood, a Sales Enablement Platform. Recognized by Entrepreneur magazine as a “mover and shaker” in sales leadership and listed by LinkedIn as one of the world’s top sales experts. Named Salesforce’s 2011 Executive of the Year by Marc Benioff after creating a Partner Relationship Management category and product line at Salesforce, Elay’s experience in B2B sales effeciencies is thoughtful and comprehensive. M.R. Rangaswami: Why is sales enablement exploding now?  Elay Cohen: There are three main factors. First, the pandemic has accelerated the need for organizations to have ... Read More

Feed: AWS Big Data Blog. The volume of data being generated globally is growing at an ever-increasing pace. Data is generated to support an increasing number of use cases, such as IoT, advertisement, gaming, security monitoring, machine learning (ML), and more. The growth of these use cases drives both volume and velocity of streaming data and requires companies to capture, processes, transform, analyze, and load the data into various data stores in near-real time. Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. As the volume of the ... Read More