Tag: streaming

Feed: All - O'Reilly Media. Author: Maliha Balala. Formation of mammatus clouds over the city of Regina Saskatchewan (source: Craig Lindsay on Wikimedia Commons) Download two free chapters from "Moving Hadoop to the Cloud" to learn more about harnessing the cloud for Hadoop clusters. When news broke that Amazon was acquiring Whole Foods, brick and mortar retailers panicked. Many industries are feeling pressure to undergo digital transformation for fear of being “Amazoned” (acquired by an online competitor). The International Data Corporation (IDC) forecasts worldwide spending on digital transformation technologies will grow to more than $1.2 trillion in 2017 (an increase ... Read More

Feed: All - O'Reilly Media. Author: Jesse Anderson. Oranges and apples (source: Ruth Hartnup on Flickr) Check out Jesse Anderson’s tutorial “Real-time systems with Spark Streaming and Kafka” at the Strata Data Conference, New York City, Sept. 25-28, 2017. Best price ends August 11. If you look up the phrase “boiling the ocean,” it’s defined as writing a post on choosing a cloud provider—there are so many different facets and use cases, and each variable complicates your choice. The key is to narrow the field to your specific situation and needs. In this article, I share some of the early ... Read More

Feed: All - O'Reilly Media. Author: Mary Treseler. Distribution Center (source: GRAPHICALBRAIN) ​This post is part of a series of Q&A interviews going behind the scenes with speakers from our October 2017 Velocity Conference in New York. Register for the O’Reilly Velocity Conference to join Joseph Breuer, Robert Reta, and other industry experts. Use code ORM20 to save 20% on your conference pass (Gold, Silver, and Bronze passes). I recently asked Joseph Breuer and Robert Reta, both Senior Software Engineers at Netflix, to discuss what they have learned through implementing a service at scale at Netflix. Joseph and Robert will ... Read More

Feed: Planet PostgreSQL. A key part of running a reliable database service is ensuring you have a good plan for disaster recovery. Disaster recovery comes into play when disks or instances fail, and you need to be able to recover your data. In those type of cases logical backups, via pg_dump, may be days old and in such cases not ideal for you to restore from. To remove the risk of data loss, many of us turn to the Postgres WAL to keep safe. Years ago Daniel Farina, now a principal engineer at Citus Data, authored a continuous archiving utility ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Venkatesan M. 1) Compare Hadoop & Spark                       Criteria                                           Hadoop                                                   Spark Dedicated storage                           HDFS                 ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Casey Stella. The Motivation Many cybersecurity problems are also big data problems.   What is more and more apparent, though, is that these problems are also problems solved by data science.  The modern cybersecurity practitioner solves cybersecurity problems by making sense of data.  This is firmly within the sweet spot where data science meets big data and reaps rewards.  The Metron team at Hortonworks is keenly aware of this relationship as we are building a cybersecurity platform atop big data infrastructure.  While there are many great platforms to use for dealing with the volume and ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. This framework based on Apache Flume, Apache Spark Streaming, and Apache Impala (incubating) can detect and report on abnormal bad HTTP requests within seconds.                      Website performance and availability are mission-critical for companies of all types and sizes, not just those with a revenue stream directly tied to the web. Web pages can become unavailable for many reasons, including overburdened backing data stores or content-management systems or a delay in load times of third-party content such as advertisements. In addition to the deterioration of user experience during such incidents, search engines quickly ... Read More

Feed: IBM Big Data & Analytics Hub - All Content; Author: siva-anne; This is final blog in the 3-part series on IBM EventStore. Part 1 introduces IBM EventStore, and part 2 describes details on ingesting event data into IBM EventStore using the scenario of Clickstream analysis for a retail business. In this blog, we will look at analyzing the Clickstream data with IBM EventStore to derive timely insights on interests of retail customers. Typically, ingesting streaming event data, persisting with low latency and analyzing it along with historical event data requires integrating multiple analytic systems. IBM EventStore is purpose built to simplify ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Kevin Jordan. We may not be able to look into the future yet, but we are at a major crossroads of data and predictive analytics that allows us to mitigate risks to prevent disasters. Whether it be an accidental house fire or a car accident, insurance companies are finding ways to create new services for their customers that avert tragedies. Big Data Forecasts When we think about normal everyday occurrences, we don’t often fixate on what would otherwise be considered mundane. The weather is usually a topic of conversation when neither party has anything ... Read More

Feed: Planet big data. Author: Meg Blanchette. August 15th, 2017 An important architectural component of any data platform is those pieces that manage data ingestion. In many of today’s “big data” environments, the data involved is at such scale in terms of throughput (think of the Twitter “firehose”) or volume (e.g., the 1000 Genomes project) that approaches and tools must be carefully considered. In the last few years, Apache Kafka and Apache Spark have become popular tools in a data architect’s tool chest, as they are equipped to handle a wide variety of data ingestion scenarios and have been used ... Read More