Tag: Cloudera

Feed: Avro – Cloudera Engineering Blog. Author: Cy Jervis. In Apache Kafka, Java applications called producers write structured messages to a Kafka cluster (made up of brokers). Similarly, Java applications called consumers read these messages from the same cluster.  In some organizations, there are different groups in charge of writing and managing the producers and consumers. In such cases, one major pain point can be in the coordination of the agreed upon message format between producers and consumers. This example demonstrates how to use Apache Avro to serialize records that are produced to Apache Kafka while allowing evolution of schemas ... Read More

Feed: Microsoft Azure Blog. Author: Rob Caron. Highlights New Azure innovation advances customer success for the cloud- and AI-powered future - In preparation for Microsoft Inspire in Las Vegas this week (where partners and Microsoft meet to connect, learn, and collaborate to accelerate the digital transformation and success of our customers), Jason Zander (Executive Vice President, Microsoft Azure) highlighted new offers and technologies that will help organizations navigate their own digital transformation and build the critical foundations for cloud and AI. He showcases momentum from businesses and partners who are investing in intelligent tech from the cloud to the edge ... Read More

Feed: All - O'Reilly Media. Author: James Furbush. Metal tubes (source: BrooklynJohn via Pixabay) Attend the Strata Data Conference to learn the skills and technologies of data engineering. As the the data space has matured, data engineering has emerged as a separate and related role that works in concert with data scientists. Ian Buss, principal solutions architect at Cloudera, notes that data scientists focus on finding new insights from a data set, while data engineers are concerned with the production readiness of that data and all that comes with it: formats, scaling, resilience, security, and more. Ryan Blue, a senior ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Igor Bobriakov. Apache Spark is a powerful open-source processing engine built around speed, ease of use, and sophisticated analytics. It has originally been developed at UC Berkeley in 2009, while Databricks was founded later by the creators of Spark in 2013. The Spark engine runs in a variety of environments, from cloud services to Hadoop or Mesos clusters. It is used to perform ETL, interactive queries (SQL), advanced analytics (e.g., machine learning) and streaming over large datasets in a wide range of data stores (e.g., HDFS, Cassandra, HBase, S3). Spark supports ... Read More

Feed: Microsoft Azure Blog. Author: Charlotte Yarkoni. Next week in Last Vegas, we will be meeting with thousands of partners from all over the world at Microsoft Inspire. The event is a forum to reinforce our commitment to making partner business opportunities the centerpiece of our investments. We continue to extend our cloud marketplace, Azure Marketplace and AppSource, capabilities to make it the best place for partners of any type to be discovered, evaluated, and ultimately have their offerings purchased and deployed by our customer base. The event also gives us the opportunity to showcase the investments we’re making to ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Cy Jervis. Cloudera Director 2.6 and Cloudera Manager 5.13 offer a simple way to have TLS configured for Cloudera Manager and CDH clusters. In this blog post, Bill Havanki describes how to use the new feature and offers technical details behind how the automatic configuration happens. An important tenet of information security is defense in depth. The idea behind defense in depth is to have multiple layers of security protecting valued assets, so that if one layer should fail, the others remain working. A real life example of defense in depth is ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Cluster admins will love the new cluster utilization reporting available in Cloudera Manager 5.7. Enterprise data hub clusters often are shared by several teams. In such multi-tenant environments, cluster administrators are required to ensure that resources are shared fairly so that one tenant cannot run jobs that starve others. To give better visibility into resource consumption in multi-tenant environments, Cloudera Manager 5.7 (in Cloudera Enterprise Flex and Data Hub Editions) has a new feature for reporting cluster utilization that provides information about overall cluster usage, as well as how YARN ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Thanks to Jonathan Natkins, a field engineer from StreamSets, for the guest post below about using StreamSets Data Collector—open source, GUI-driven ingest technology for developing and operating data pipelines with a minimum of code—and Cloudera Search and HUE to build a real-time search environment. As pressure mounts on data engineers to deliver more data from more sources in less time, StreamSets Data Collector can serve as a linchpin in the data management process, helping them simplify ingest pipeline development and operations across the rapidly evolving ecosystem of big data tools ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Unique across all options, Cloudera Manager makes it easy to do what would otherwise be a disruptive operation for operators and users. For the increasing number of customers that rely on enterprise data hubs (EDHs) for business-critical applications, it is imperative to minimize or eliminate downtime — thus, Cloudera has focused intently on making software upgrades a routine, non-disruptive operation for EDH administrators and users. With Cloudera Manager 4.6 and later, it is extremely easy to do minor version upgrades (from CDH 4.3 to CDH 4.6) with zero downtime, and ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Thanks to Bill Podell, VP Big Data and BI Practice, MBI Solutions, for the guest post below. Capacity planning has long been a critical component of successful implementations for production systems. Today, Big Data calls for a particularly deep understanding of capacity management – because resource utilization explodes as business users, analysts, and data scientists jump onboard to analyze and use newly found data. The resource impact can escalate very quickly, causing poor loading and or response times. The result is throwing more hardware at the issue without any understanding ... Read More