Category: Cloudera

Feed: Avro – Cloudera Engineering Blog. Author: Cy Jervis. In Apache Kafka, Java applications called producers write structured messages to a Kafka cluster (made up of brokers). Similarly, Java applications called consumers read these messages from the same cluster.  In some organizations, there are different groups in charge of writing and managing the producers and consumers. In such cases, one major pain point can be in the coordination of the agreed upon message format between producers and consumers. This example demonstrates how to use Apache Avro to serialize records that are produced to Apache Kafka while allowing evolution of schemas ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Cy Jervis. Cloudera Director 2.6 and Cloudera Manager 5.13 offer a simple way to have TLS configured for Cloudera Manager and CDH clusters. In this blog post, Bill Havanki describes how to use the new feature and offers technical details behind how the automatic configuration happens. An important tenet of information security is defense in depth. The idea behind defense in depth is to have multiple layers of security protecting valued assets, so that if one layer should fail, the others remain working. A real life example of defense in depth is ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Cluster admins will love the new cluster utilization reporting available in Cloudera Manager 5.7. Enterprise data hub clusters often are shared by several teams. In such multi-tenant environments, cluster administrators are required to ensure that resources are shared fairly so that one tenant cannot run jobs that starve others. To give better visibility into resource consumption in multi-tenant environments, Cloudera Manager 5.7 (in Cloudera Enterprise Flex and Data Hub Editions) has a new feature for reporting cluster utilization that provides information about overall cluster usage, as well as how YARN ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Thanks to Jonathan Natkins, a field engineer from StreamSets, for the guest post below about using StreamSets Data Collector—open source, GUI-driven ingest technology for developing and operating data pipelines with a minimum of code—and Cloudera Search and HUE to build a real-time search environment. As pressure mounts on data engineers to deliver more data from more sources in less time, StreamSets Data Collector can serve as a linchpin in the data management process, helping them simplify ingest pipeline development and operations across the rapidly evolving ecosystem of big data tools ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Unique across all options, Cloudera Manager makes it easy to do what would otherwise be a disruptive operation for operators and users. For the increasing number of customers that rely on enterprise data hubs (EDHs) for business-critical applications, it is imperative to minimize or eliminate downtime — thus, Cloudera has focused intently on making software upgrades a routine, non-disruptive operation for EDH administrators and users. With Cloudera Manager 4.6 and later, it is extremely easy to do minor version upgrades (from CDH 4.3 to CDH 4.6) with zero downtime, and ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Thanks to Bill Podell, VP Big Data and BI Practice, MBI Solutions, for the guest post below. Capacity planning has long been a critical component of successful implementations for production systems. Today, Big Data calls for a particularly deep understanding of capacity management – because resource utilization explodes as business users, analysts, and data scientists jump onboard to analyze and use newly found data. The resource impact can escalate very quickly, causing poor loading and or response times. The result is throwing more hardware at the issue without any understanding ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Thanks to Jonathan Natkins of WibiData for the post below about how his company extended Cloudera Manager to manage Kiji. Learn more about Kiji and the organizations using it to build real-time HBase applications at Kiji Sessions, happening on May 6, 2014, the day after HBaseCon. As a partner of Cloudera, WibiData sees Cloudera Manager’s new extensibility framework as one of the most exciting parts of Cloudera Enterprise 5. Cloudera Manager 5.0.0 provides the single-pane view that Apache Hadoop administrators and operators want to effectively manage a cluster of machines ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. One instance of Cloudera Manager (CM) can manage N clusters. In the current Role Based Access Control (RBAC) model, CM users hold privileges and permissions across everything in CM’s purview (including every cluster that CM manages). For example, Read-Only user John is a user who can perform all the actions of Read-Only users on all clusters managed by CM. The “Cluster Admin” user Chris is a cluster administrator of all the clusters managed by CM. This RBAC model has been sufficient to satisfy a number of use-cases. However, as the number ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. More and more customers are using automation/configuration management frameworks alongside Cloudera Manager. As Apache Hadoop clusters continue to grow in size, complexity, and business importance as the foundational infrastructure for an Enterprise Data Hub, the use cases for a robust and mature management console expand.  As those clusters become larger and more complex, many operators are looking to use configuration management/automation frameworks like Ansible, Chef, and Puppet alongside Cloudera Manager to help automate the deployment and configuration process. (Others are rolling their own via the open source Cloudera Manager REST API.) Here ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. Cloudera Manager 4.7 added support for managing Cloudera Search 1.. Thus Cloudera Manager users can easily deploy all components of Cloudera Search (including Apache Solr) and manage all related services, just like every other service included in CDH (Cloudera’s distribution of Apache Hadoop and related projects). In this how-to, you will learn the steps involved in adding Cloudera Search to a Cloudera Enterprise (CDH + Cloudera Manager) cluster. Installing the SOLR Parcel In our example, the cluster uses a CDH 4.4 parcel and is running Apache ZooKeeper, HDFS, and Apache ... Read More