Tag: Hive

Feed: Microsoft Azure Blog. Author: Bhanu Prakash. Azure HDInsight is the only fully-managed cloud Hadoop & Spark offering that gives you optimized open-source analytic clusters for Spark, Hive, MapReduce, HBase, Storm, Kafka, and Microsoft R Server backed by a 99.9% SLA. You can deploy these big data technologies and ISV applications as managed clusters with enterprise-level security and monitoring. Given the broad scope of open source technologies on big data, we have received many requests for a detailed guide around HDInsight. This post outlines the key areas covered by the guide and also outlines some other options for learning on ... Read More

Feed: All - O'Reilly Media. Author: Lu Niu, Peng Du, Zhenxiao Luo. Uber’s Presto architecture (source: Courtesy of Zhenxiao Luo) For more on efficient geospatial analysis, check out Zhenxiao Luo and Wei Yan’s session, “Geospatial big data analysis at Uber,” at the Strata NYC Data Conference in NYC, September 25-28, 2017. From determining the most convenient rider pickup points to predicting the fastest routes, Uber aims to use data-driven analytics to create seamless trip experiences. Within engineering, analytics inform decision-making processes across the board. One of the distinct challenges for Uber is analyzing geospatial big data. City locations, trips, and ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. There are several ingestion techniques that can be used to add data to Big SQL and Hive. This blog will give an overview of the various flavors of adding files or appending to files already in HDFS and the commands to execute from either Hive or Big SQL. The technique of adding files directly to HDFS can offer one of the fastest ways to ingest data into Hadoop. Create External Hadoop Tables If files already reside on HDFS, Big SQL tables can be created with the location of these files specified. These tables are ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Will Xu. As more and more workloads are being brought onto modern hardware in the cloud, it’s important for us to understand how to pick the best databases that can leverage the best hardware. Amazon has introduced instances with directly attached SSD (Solid state drive). Both Apache HBase and Apache Cassandra are popular key-value databases. In this benchmark, we hope to learn more about how they leverage the directly attached SSD in a cloud environment. The benchmark is designed for running Apache HBase and Apache Cassandra in an optimal production environment. This means using ... Read More

Feed: All - O'Reilly Media. Author: Tobi Bosede. Wrong Way Go Back (source: Johnny Jet on Flickr) Check out Tobi Bosede’s session “Big data analysis of futures trades” at the Strata Data Conference in New York City, September 25-28, 2017, for more tips on how to work efficiently in Spark, and how it can be used in predicting trade volume. Registration is now open. My upcoming Strata Data NYC 2017 talk about big data analysis of futures trades is based on research done under the limited funding conditions of academia. This meant that I did not have an infrastructure team, ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Piet Loubser. The annual Strata Data Conference will make its next stop in New York the week of September 25-29. The core theme of the conference this year is around driving business transformations through the power of data. And in the world of data, few topics excite as much as data science, machine learning, and deep learning. And with good reasons! The combination of the Hadoop big data world and the much older data science, machine learning world is a perfect marriage. The Big Data Journey While Apache Hadoop has been around for a ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. When YARN is enabled for Big SQL, YARN controls how much memory and CPU resources are assigned to Big SQL at any given point in time. Other applications such as Spark or Hive can continue to use YARN resources when YARN is enabled for Big SQL. Read more about Big SQL and YARN integration. There are several YARN and Big SQL configuration parameters that are at play with YARN is enabled with Big SQL. These configuration parameters and best practice to setting these parameter are discussed in this blog. YARN Container Memory Settings The ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. YARN is a central resource manager component which manages and allocates resources to applications running on Hadoop. Multiple applications can run on Hadoop and they can share a common resource manager via YARN. Apache Slider allows for non-YARN services to be managed by YARN. Big SQL is integrated with YARN through Apache Slider since Big SQL 4.2.5 (improvements have been made in Big SQL 5.0.1 in this area). When a component requests resources from YARN, it is put on a YARN priority queue and YARN schedules the allocation based on priority. When the request ... Read More

Feed: Planet big data. Author: Ronald van Loon.   Leveraging the use of big data, as an insight-generating engine, has driven the demand for data scientists at enterprise-level, across all industry verticals. Whether it is to refine the process of product development, help improve customer retention, or mine through the data to find new business opportunities—organizations are increasingly relying on the expertize of data scientists to sustain, grow, and outdo their competition. Consequently, as the demand for data scientists increase, the discipline presents an enticing career path for students and existing professionals. This includes those who are not data scientists ... Read More

Feed: SQL Server Customer Advisory Team. Author: Murshed Zaman_AzureCAT. Reviewed by: John Hoang, Dimitri Furman, Mike Weiner, Sanjay Mishra We often get questions from customers related to Azure SQL DW being a good fit for moving a particular database to the cloud. Some of the typical questions are: I have a database on-premises. I would like to move it into the cloud because that seems to be the future direction. Can I use Azure SQL DW for it? I am running out of storage space on my on-premises system. I would like to move my database to the cloud. Is ... Read More