Tag: partitions

Feed: Cloudera Engineering Blog » Hadoop. Author: Alex Kozlov. Apache Hadoop and Apache HBase are gaining popularity due to their flexibility and tremendous work that has been done to simplify their installation and use.  This blog is to provide guidance in sizing your first Hadoop/HBase cluster.  First, there are significant differences in Hadoop and HBase usage.  Hadoop MapReduce is primarily an analytic tool to run analytic and data extraction queries over all of your data, or at least a significant portion of them (data is a plural of datum).  HBase is much better for real-time read/write/modify access to tabular data.  Both ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Arvind Prabhakar. Hadoop has emerged as an indispensable component of any data-intensive enterprise infrastructure.  In many ways, working with large datasets on a distributed computing platform (powered by commodity hardware or cloud infrastructure) has never been easier. But because customers are running clusters consisting of hundreds or thousands of nodes, and are processing massive quantities of data from production systems every hour, the logistics of efficient platform utilization can quickly become overwhelming. To deal with this challenge, the Yahoo! engineering team created Oozie – the Hadoop workflow engine. We are pleased to provide ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Amr Awadallah. At Cloudera, we frequently work with leading Hadoop developers to produce guest blog posts of general interest to the community. We started a project with Pete Skomoroch a while back, and we were so impressed with his work, we’ve decided to bring Pete on as a regular guest blogger. Pete can show you how to do some pretty amazing things with Hadoop, Pig and Hive and has a particular bias towards Amazon EC2. With that, I’m happy to welcome Pete to the blog, and hope you enjoy his first post as much ... Read More

Feed: AWS Big Data Blog. Amazon Redshift Spectrum enables you to run Amazon Redshift SQL queries against data that is stored in Amazon S3. With Amazon Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond the data that is stored on local disks in your data warehouse. You can query vast amounts of data in your Amazon S3 “data lake” without having to go through a tedious and time-consuming extract, transfer, and load (ETL) process. Amazon Redshift Spectrum applies sophisticated query optimization and scales processing across thousands of nodes to deliver fast performance.In this blog post, we have ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Tom White. This piece is based on the talk “Practical MapReduce” that I gave at Hadoop User Group UK on April 14. 1. Use an appropriate MapReduce language There are many languages and frameworks that sit on top of MapReduce, so it’s worth thinking up-front which one to use for a particular problem. There is no one-size-fits-all language; each has different strengths and weaknesses. Java: Good for: speed; control; binary data; working with existing Java or MapReduce libraries. Pipes: Good for: working with existing C++ libraries. Streaming: Good for: writing MapReduce programs in ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Tom White. The first release (0.19.0) from the 0.19 branch of Apache Hadoop Core was made on November 24. Many changes go into a release like this, and it can be difficult to get a feel for the more significant ones, even with the detailed Jira log, change log, and release notes. (There’s also JDiff documentation, which is a great way to see how the public API changed, via a JavaDoc-like interface.) This post gives a high-level feel for what’s new. Core Hadoop Core now requires Java 6 to run (HADOOP-2325). Most installations are ... Read More

Feed: Cloudera Engineering Blog » CDH. Author: Justin Kestelyn. Cloudera Enterprise 5.5 (comprising CDH 5.5, Cloudera Manager 5.5, and Cloudera Navigator 2.4) has been released. Cloudera is excited to bring you news of Cloudera Enterprise 5.5. Our persistent emphasis on quality is especially pronounced in this release, with more than 500 issues identified and triaged during its development. A highlight of this release is the inclusion of Cloudera Navigator Optimizer (available in limited beta for select Cloudera Enterprise customers; learn more here), a new Web-based service based on previous Xplain.io technology that provides ETL consolidation, BI optimization, query de-duplication, and ... Read More

Feed: Cloudera Engineering Blog » CDH. Author: Justin Kestelyn. This post contains answers to common questions about deploying and configuring Apache Kafka as part of a Cloudera-powered enterprise data hub. Cloudera added support for Apache Kafka, the open standard for streaming data, in February 2015 after its brief incubation period in Cloudera Labs. Apache Kafka now is an integrated part of CDH, manageable via Cloudera Manager, and we are witnessing rapid adoption of Kafka across our customer base. As more customers adopt Apache Kafka, a common set of questions from them about development and deployment have emerged as a pattern ... Read More

Feed: Hazelcast Blog. Author: Christoph Engelbert. We had the chance to speak with Ben Evans, Java Champion, author for O’Reilly and InfoQ, as well as co-founder of JClarity, a startup which delivers performance tools and services to help development and ops teams. Together with his recent whitepaper on Hazelcast in combination with Spark, he also delivered a small application, BetLeopard, to demonstrate the integration between the two technologies. Chris Engelbert: You’re around in the Java world for quite some time, when did you start with Java and why? Ben Evans: Well, one of the things about me is that I ... Read More

Feed: Hazelcast Blog. Author: Hazelcast Community. DataMountaineer recently published a blog post entitled, “Hazelcast – Low Latency Datastores for IOT“. In the post, DataMountaineer presents a strategy for building low latency datastores for Internet of Things using Kafka and Hazelcast. The Internet of Things is on the rise, it was certainly a buzzword of 2016. Gartner thinks so, they say there will be 20 billion devices online by 2020 with all of them transmitting (streaming) data. These devices are not limited to new devices, more and more we are seeing our clients want to connect into manufacturing control systems such ... Read More