Tag: Hive

Feed: Microsoft Azure Blog. Author: James Baker. On February 7, 2019 we announced the general availability of Azure Data Lake Storage (ADLS) Gen2. Azure is now the only cloud provider to offer a no-compromise cloud storage solution that is fast, secure, massively scalable, cost-effective, and fully capable of running the most demanding production workloads. In this blog post we’ll take a closer look at the technical foundation of ADLS that will power the end to end analytics scenarios our customers demand. ADLS is the only cloud storage service that is purpose-built for big data analytics. It is designed to integrate ... Read More

Feed: What's New. Now customers can customize the Amazon S3 prefix using the Apache Hive format and timestamp formats (supported by the Java DateTimeFormatter class).   For a list of regions where Amazon Kinesis Data Firehose is available, see the AWS Region Table ... Read More

Feed: Cloudera Engineering Blog. Author: Tom Wheeler. [Editor’s note: This article was originally published on the Hortonworks Community Connection, but reproduced here because CDSW is now available on both Cloudera and Hortonworks platforms.] Using Deployed Models as a Function as a Service    Using Cloudera Data Science Workbench with Apache NiFi, we can easily call functions within our deployed models from Apache NiFi as part of flows. I am working against CDSW on HDP (https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_hdp.html),  but it will work for all CDSW regardless of install type. In my simple example, I built a Python model that uses TextBlob to run sentiment analysis against ... Read More

Feed: Cloudera Engineering Blog. Author: Tom Wheeler. Cloudera Data Warehouse offers a powerful combination of flexibility and cost-savings. Using Cloudera Data Warehouse, you can transform and optimize your current traditional data warehouse by moving select workloads to your CDH cluster. This article shows you how to transform your current setup into a modern data warehouse by moving some initial data over to Impala on your CDH cluster. Prerequisites To use the following data import scenario, you need the following: A moderate-sized CDH cluster that is managed by Cloudera Manager, which configures everything properly so you don’t need to worry about ... Read More

Feed: Cloudera Engineering Blog. Author: Tom Wheeler. In the last few years, IT security threats to enterprise systems have increased, which has necessitated installing log ingestion and analysis solutions in any enterprise network. This blog post illustrates how Cloudera built its own scalable solution for log ingestion and analytics using Apache Spot and Cloudera Altus. By leveraging transient workloads in the cloud, Cloudera reduced the solution’s operational costs by 50% when compared to traditional, persistent cluster approaches. At Cloudera, the Infosec team needed to build a centralized log ingestion and analytical solution to monitor all key systems in the company ... Read More

Feed: IBM Big Data Hub Blog; Author: kevin-oliver; The focus on customer needs for greater choice and flexibility is a constant at the IBM Think 2019 conference. Nowhere is this more evident than in IBM Hybrid Data Management, which supports data of any type, source and structure, be it on-premises or in the cloud.On Monday, the level of customer choice expanded even further with two key announcements, including IBM Db2 Warehouse on Cloud, now available on Amazon Web Services (AWS), and the introduction of a cost optimized, symmetric multiprocessing (SMP) elastic scaling configuration. Both announcements are great news for organizations ... Read More

Feed: Planet big data. Author: David Smith. A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so. Open Source AI, ML & Data Science News ONNX, the open interchange format for AI models, updates to version 1.4.0 with support for models larger than 2Gb. PT-BERT, a PyTorch implementation of Google's BERT language representation model, adds new pre-trained language models: GPT and Transformer-XL. Stanford University has released StanfordNLP, a natural language ... Read More

Feed: AWS Big Data Blog. Clickstream events are small pieces of data that are generated continuously with high speed and volume. Often, clickstream events are generated by user actions, and it is useful to analyze them.For example, you can detect user behavior in a website or application by analyzing the sequence of clicks a user makes, the amount of time the user spends, where they usually begin the navigation, and how it ends. By tracking this user behavior in real time, you can update recommendations, perform advanced A/B testing, push notifications based on session length, and much more. To track ... Read More

Feed: What's New. The AWS Glue Data Catalog is a fully managed, Apache Hive Metastore compatible, metadata repository. Customers can use the Data Catalog as a central repository to store structural and operational metadata for their data. AWS Glue provides out-of-box integration with Amazon EMR that enables customers to use the AWS Glue Data Catalog as a replacement for an external Hive Metastore. To learn more, visit our documentation. This code serves as a reference implementation for building a Hive Metastore compatible client that connects to the AWS Glue Data Catalog. With this release, customers and partners can build custom ... Read More

Feed: AWS Big Data Blog. With the ever-evolving and growing role of data in today’s world, data governance is an essential aspect of effective data management. Many organizations use a data lake as a single repository to store data that is in various formats and belongs to a business entity of the organization. The use of metadata, cataloging, and data lineage is key for effective use of the lake.This post walks you through how Apache Atlas installed on Amazon EMR can provide capability for doing this. You can use this setup to dynamically classify data and view the lineage of ... Read More