Tag: Hadoop

Feed: R-bloggers. Author: Peter's stats stuff - R. I recently read the high quality book Global Inequality by Branko Milanovic. When reading this sort of thing, I often find I can increase my engagement with a topic by playing around with the data myself. In this day and age, this is much more likely to be possible than a couple of decades back! I remember when I first studied development economics, typing into Lotus 123 the data from tables in the back of a World Development Report. These days to get the same data we just fire up R and ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Justin Kestelyn. Welcome to our fourth edition of “This Month in the Ecosystem,” a digest of highlights from October 2013 (never intended to be comprehensive; for completeness, see Hadoop Weekly). For generating sheer excitement, that month installed a high bar to meet in the future: Hadoop 2 Became Stable with the 2.2 (“GA”) ReleaseThe idea that this news could be done justice in a few sentences strains credulity. (Furthermore, it has been covered elsewhere widely and thoroughly at this point.) Suffice to say here that this release will transform the Big Data ecosystem ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Sandy Ryza. Cloudera Manager lets you add a YARN service in the same way you would add any other Cloudera Manager-managed service. In Apache Hadoop 2, YARN and MapReduce 2 (MR2) are long-needed upgrades for scheduling, resource management, and execution in Hadoop. At their core, the improvements separate cluster resource management capabilities from MapReduce-specific logic. They enable Hadoop to share resources dynamically between MapReduce and other parallel processing frameworks, such as Cloudera Impala; allow more sensible and finer-grained resource configuration for better cluster utilization; and permit Hadoop to scale to accommodate more and ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Sandy Ryza. In Apache Hadoop 2, YARN and MapReduce 2 (MR2) are long-needed upgrades for scheduling, resource management, and execution in Hadoop. At their core, the improvements separate cluster resource management capabilities from MapReduce-specific logic. They enable Hadoop to share resources dynamically between MapReduce and other parallel processing frameworks, such as Cloudera Impala; allow more sensible and finer-grained resource configuration for better cluster utilization; and permit Hadoop to scale to accommodate more and larger jobs. In this post, users of CDH (Cloudera’s distribution of Hadoop and related projects) who program MapReduce jobs will ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Justin Kestelyn. In software development, there is no substitute for having choices. Furthermore, freedom of choice – between frameworks, APIs, and languages — is a major fuel source for platform adoption across any successful ecosystem. In the case of development on CDH, the open source core of Cloudera’s Big Data platform containing Apache Hadoop and related ecosystem projects, the choices have expanded dramatically in the past three weeks: Spark + CDH Cloudera has announced direct support for Apache Spark (incubating) with CDH. Spark, the in-memory data processing framework designed at UC Berkeley’s AMPLab ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Justin Kestelyn. We are just a weekend away from the Biggest. Strata + Hadoop World. Ever. Not to make non-attendees feel bad — the show sold out a couple weeks ago — but some great things are in store for technical end-users (as well as everyone else): Hear keynotes from Mike Olson and Doug Cutting: Be prepared to hear two of the most influential voices in Big Data make some Big News.  Learn “How to Develop a Hadoop Data Application” (via the CDK) from Tom White, Eric Sammer, and Joey Echeverria. No pressure! The ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Rob Weltman. There are a number of special “users” with roles to play in the Apache Hadoop environment. For your reference, we have summarized them below as of CDH 4.4. Kerberos principals (used for authentication in a secure cluster) are not covered here. The specific user IDs listed are the ones created by default on installation but they are configurable unless otherwise indicated. Project User Group Notes HDFS hdfs hdfs The NameNode and DataNodes run as this user, and the HDFS root directory as well as the directories used for edit logs should ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Aaron Myers. The release of Apache Hadoop 2, as announced today by the Apache Software Foundation, is an exciting one for the entire Hadoop ecosystem. Cloudera engineers have been working hard for many months with the rest of the vast Hadoop community to ensure that Hadoop 2 is the best it can possibly be, for the users of Cloudera’s platform as well as all Hadoop users generally. Hadoop 2 contains many major advances, including (but not limited to): High availability for the HDFS NameNode, which eliminates the previous SPOF in HDFS. Support for ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Justin Kestelyn. In a fast-moving project like Apache Hadoop, there are always exciting new features introduced in each release. While it is tempting to make the most of these new features by upgrading to the latest release, users are often concerned about their code continuing to run. In this post, you’ll get an overview of the the Hadoop API annotations and compatibility policies. Hadoop annotates specific APIs to be safe for use by end-users. By using these APIs, users can ensure their code works across a set of releases and be aware of ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Karina Babcock. The Cloudera Sessions fall series is coming to a close next week, but first we’ll make a final stop in Milwaukee, Wisconsin (on Oct. 17), where attendees will hear about edo — a company that is revolutionizing the advertising space by closing the loop between promotions and point-of-sale transactions. In Milwaukee, edo CTO Jeff Sippel will engage in a fireside chat with Cloudera’s VP of marketing, Alan Saldich. At edo, Jeff is responsible for the strategy, planning, and execution for the systems — including Apache Hadoop — that power the edo offer platforms ... Read More