Tag: streaming

Feed: Cloudera Engineering Blog » Hadoop. Author: Jon Zuanich. Guest re-post from Phil Whelan, a large-scale web-services consultant based in Vancouver, BC. Here I demonstrate, with repeatable steps, how to fire-up a Hadoop cluster on Amazon EC2, load data onto the HDFS (Hadoop Distributed File-System), write map-reduce scripts in Ruby and use them to run a map-reduce job on your Hadoop cluster. You will not need to ssh into the cluster, as all tasks are run from your local machine. Below I am using my MacBook Pro as my local machine, but the steps I have provided should be reproducible on ... Read More

Feed: Big Data Made Simple - One source. Many perspectives.. Author: Baiju NT. Big Data can be intimidating! If you are new to Big Data, please read ‘What is Big Data’, ‘Who coined Big Data’ to get you started. With the basic concepts under your belt, let’s focus on some key terms to impress your date or boss or family. By the way, I am putting together a much more exhaustive list of Big Data terms (almost 100) and if you are interested in that, please leave a comment below with a note ‘I want more of Big data’ or ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Tom Hastain. We’ve just published our most recent customer case study! This one comes from straight from Japan and is emblematic of the trend towards connected data architectures throughout the North Asian region, and Hortonworks’ continued growth in the same. Mitsubishi Fuso Truck and Bus Corporation (Mitsubishi Fuso) is a leading manufacturer of trucks, buses, and industrial engines. The company has embarked on a massive Big Data transformation to capitalize on the value of large data sets. Included is the sensor data streaming from its vehicles. These sensors allow for tele-diagnosis of vehicle problems, ... Read More

Feed: Planet big data. Author: Ronald van Loon. I attended the Huawei European Innovation Day recently, and was enthralled by how the new technology is giving rise to industrial revolutions. These revolutions are what will eventually unlock the development potential around the world. It is important to leverage the emerging technologies, since they are the resources which will lead us to innovation and progress. Huawei is innovative in its partnerships and collaboration to define the future, and the event was a huge success. For many people, the concept of Artificial Intelligence (AI) is a thing of the future. It is the technology that has ... Read More

Feed: Google Cloud Big Data and Machine Learning Blog. Author: Google Cloud Big Data and Machine Learning Blog Team. Innovation in data processing and machine learning technology Tuesday, June 27, 2017 By Tudor Marian, Marián Dvorský and Ashwin Kumar, Software Engineers and Sergei Sokolenko, Product Manager Learn how the new service-based Shuffle feature brings significant performance improvements to your Cloud Dataflow pipelines. Google Cloud Platform (GCP) has a track record of continually bringing innovations in data processing, such as streaming and Apache Beam-based SDKs, to Cloud Dataflow customers. Today, we're introducing a new feature (currently in beta) that significantly improves ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Jon Zuanich. Neil Kodner, an independent consultant, is the guest author of this post. Neil found inspiration, which spurred innovation at Hadoop World 2010 from a moments decision to capture the #hw2010 streaming Twitter feed. During the Hadoop World 2010 keynote, a majority of attendees were typing away on their laptops as Mike Olson and Tim O’Reilly dazzled the audience.  Many of these laptop-users appeared to be tweeting as the keynote was taking place.   Since I have more than a passing interest in twitter, Hadoop, and text mining, I thought it would be a great ... Read More

Feed: Google Cloud Big Data and Machine Learning Blog. Author: Google Cloud Big Data and Machine Learning Blog Team. Innovation in data processing and machine learning technology Monday, June 26, 2017 By Jibran Saithi, Lead Architect, Qubit At Qubit, we enable over 400 million personalized experiences per month, and to power those experiences we process data from websites, mobile apps and other customer touchpoints. Managing and processing this data in near-real time is a significant challenge; various stages of the pipeline process events at over 500K messages per second, all of which need to be validated, enriched and routed with ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Tom White. Apache Hadoop 0.21.0 was released on August 23, 2010. The last major release was 0.20.0 in April last year, so it’s not surprising that there are so many changes in this release, given the amount of activity in the Hadoop development community. In fact, there were over 1300 issues fixed in JIRA (Common, HDFS, MapReduce), the issue tracker used for Apache Hadoop development. Bear in mind that the 0.21.0 release, like all dot zero releases, isn’t suitable for production use. With such a large delta from the last release, it is ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Arvind Prabhakar. Hadoop has emerged as an indispensable component of any data-intensive enterprise infrastructure.  In many ways, working with large datasets on a distributed computing platform (powered by commodity hardware or cloud infrastructure) has never been easier. But because customers are running clusters consisting of hundreds or thousands of nodes, and are processing massive quantities of data from production systems every hour, the logistics of efficient platform utilization can quickly become overwhelming. To deal with this challenge, the Yahoo! engineering team created Oozie – the Hadoop workflow engine. We are pleased to provide ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Amr Awadallah. (guest blog post by Pete Skomoroch) In a previous post, I outlined how to build a basic trend tracking site called trendingtopics.org with Cloudera’s Distribution for Hadoop and Hive.  TrendingTopics uses Hadoop to identify the top articles trending on Wikipedia and displays related news stories and charts.  The data powering the site was pulled from an Amazon EBS Wikipedia Public Dataset containing 8 months of hourly pageview logfiles.  In addition to the pageview logs, the EBS data volume also includes the full text content and link graph for all articles.  This ... Read More