Tag: storage

Feed: Big Data Made Simple - One source. Many perspectives.. Author: Baiju NT. In a country that has just witnessed the latest epidemic in modern-day cyberattacks, namely, the National Health Service being compromised; data security is fast reaching the forefront of corporate mindsets. The truth is that no server is impenetrable and no data is unreachable for the most sophisticated of hackers, the perfect example of this being the 1 billion Yahoo accounts that were at risk from the breach reported last year. However, groups like WannaCry – the North Korean hackers linked to the NHS attack could have been ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Lars George. This post is a follow up to an earlier one called Apache Hadoop Log Files: Where to find them in CDH, and what info they contain. It lists nicely the various places Hadoop uses to store log and other info files while it is running. Over time we have seen changes though in different Hadoop and CDH releases that affect where files are stored or how long they are retained. Below strives to document the status quo (no, not the band) of all the log and info files found while running ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Jon Zuanich. Continued Guest Post from Michael Schatz and Jimmy Lin Part 2: Efficient Graph Analysis in Hadoop with Schimmy In part 1, we looked at how extremely large graphs can be represented and analyzed in Apache Hadoop/MapReduce. Here in part 2 we will examine this design in more depth to identify inefficiencies, and present some simple solutions that can be applied to many Hadoop/MapReduce graph algorithms. The speedup using these techniques is substantial: as a prototypical example, we were able to reduce the running time of PageRank on a webgraph with 50.2 ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Jon Zuanich. This is a guest post provided by Booz Allen Hamilton data analysis consultant, Aaron Cordova.  Aaron specializes in large-scale distributed data processing systems. Working within the U.S. federal government arena provides plenty of opportunities to encounter large-scale data analysis. Projects ranging from massive health studies to high-velocity network security events to new sources of imagery and video represent a huge increase in the amount of data that must be not only stored but processed quickly and efficiently. These challenges are at once a daunting and exciting chance to turn data into ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Jon Zuanich. Hadoop is increasingly being adopted by many Fortune 500 enterprises. Some of the speakers featured at Hadoop World this year include leading companies who have been able to create new value for their business using Hadoop. The presentations at Hadoop World are focused on how Hadoop is solving business problems for these enterprises.  Below are three examples of leading enterprises that will present how Hadoop has impacted their businesses. GE, Product Manager, Linden Hillenbrand, will be talking about how Hadoop has improved GE’s Marketing & Communications functions.  One capability GE has ... Read More

Feed: Google Cloud Big Data and Machine Learning Blog. Author: Google Cloud Big Data and Machine Learning Blog Team. Innovation in data processing and machine learning technology Monday, June 26, 2017 By Jibran Saithi, Lead Architect, Qubit At Qubit, we enable over 400 million personalized experiences per month, and to power those experiences we process data from websites, mobile apps and other customer touchpoints. Managing and processing this data in near-real time is a significant challenge; various stages of the pipeline process events at over 500K messages per second, all of which need to be validated, enriched and routed with ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Tom White. Apache Hadoop 0.21.0 was released on August 23, 2010. The last major release was 0.20.0 in April last year, so it’s not surprising that there are so many changes in this release, given the amount of activity in the Hadoop development community. In fact, there were over 1300 issues fixed in JIRA (Common, HDFS, MapReduce), the issue tracker used for Apache Hadoop development. Bear in mind that the 0.21.0 release, like all dot zero releases, isn’t suitable for production use. With such a large delta from the last release, it is ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: John Kreisa. This post was contributed by Jonathan Seidman from Orbitz. Jonathan is a Lead Engineer on the Intelligent Marketplace/Machine Learning team at Orbitz Worldwide . You can hear more from Jonathan at Hadoop World October 12th in NYC. Orbitz Worldwide (NYSE:OWW) is composed of a global portfolio of online consumer travel brands including Orbitz, Cheaptickets, The Away Network, ebookers and HotelClub, Additionally, the company operates business-to-business service: Orbitz Worldwide Distribution provides third parties such as Amtrak, Delta, LAN, KLM, Air France and a number of other leading airlines hotel booking capabilities, and Orbitz for ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Alex Kozlov. Apache Hadoop and Apache HBase are gaining popularity due to their flexibility and tremendous work that has been done to simplify their installation and use.  This blog is to provide guidance in sizing your first Hadoop/HBase cluster.  First, there are significant differences in Hadoop and HBase usage.  Hadoop MapReduce is primarily an analytic tool to run analytic and data extraction queries over all of your data, or at least a significant portion of them (data is a plural of datum).  HBase is much better for real-time read/write/modify access to tabular data.  Both ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Aaron Kimball. Cloudera customers usually have two major sources of data: log files, which can be imported to Hadoop via Flume, and relational databases. Throughout the previous releases of CDH2 and CDH3, Cloudera has included a package we’ve developed called Sqoop. Sqoop can perform batch imports and exports between relational databases and Hadoop, storing data in HDFS and creating Hive tables to hold results. We described its motivation and some use cases in a previous blog post a while ago. In CDH3b2, we’ve included a greatly-expanded version of Sqoop which has had a ... Read More