Tag: workflow

Feed: Cloudera Engineering Blog » Hadoop. Author: Aaron Kimball. Cloudera customers usually have two major sources of data: log files, which can be imported to Hadoop via Flume, and relational databases. Throughout the previous releases of CDH2 and CDH3, Cloudera has included a package we’ve developed called Sqoop. Sqoop can perform batch imports and exports between relational databases and Hadoop, storing data in HDFS and creating Hive tables to hold results. We described its motivation and some use cases in a previous blog post a while ago. In CDH3b2, we’ve included a greatly-expanded version of Sqoop which has had a ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Arvind Prabhakar. Hadoop has emerged as an indispensable component of any data-intensive enterprise infrastructure.  In many ways, working with large datasets on a distributed computing platform (powered by commodity hardware or cloud infrastructure) has never been easier. But because customers are running clusters consisting of hundreds or thousands of nodes, and are processing massive quantities of data from production systems every hour, the logistics of efficient platform utilization can quickly become overwhelming. To deal with this challenge, the Yahoo! engineering team created Oozie – the Hadoop workflow engine. We are pleased to provide ... Read More

Feed: Alation. Author: Stephanie McReynolds. Data lakes, while useful in helping you to capture all of your data, are only the first step in extracting the value of that data. Self-service analytics tools like Tableau help by democratizing the data analysis process, causing disruption to existing Extract, Transform, and Load (ETL) tools from the likes of Informatica and IBM Datastage. Adopting a self-service analytics solution is a step in the right direction, but most companies we encounter still haven’t solved the problem of how to make data easily accessible for a non-technical user.We recently announced an integration with Trifacta to ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Matt Massie. We asked Brian Bockelman, a Post Doc Research Associate in the Computer Science & Engineering Department at the University of Nebraska–Lincoln, to tell us how Hadoop is being used to process the results from High-Energy Physics experiments.  His response gives insights into the kind and volume of data that High-Energy Physics experiments generate and how Hadoop is being used at the University of Nebraska. -Matt In the least technical language, most High Energy Physics (HEP) experiments involve colliding highly energetic, common particles together in order to create small, exotic, and incredibly ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Ian Scofield. By Ian Scofield. Ian is a Partner Solutions Architect (SA) at AWS. Over the course of three blog posts (Parts 1, 2, and 3), we’ve been discussing a new way for APN Partners to onboard customers, focusing on the creation of cross-account roles.  Throughout this series, we’ve proposed the usage of AWS CloudFormation for the creation of this role, to provide an automated process that can seamlessly integrate into an existing onboarding workflow. In this post, we’ll recap the components discussed so far, put it all together, and show you the completed ... Read More

Feed: Percona Database Performance Blog. Author: Tim Vaillancourt. Tim Vaillancourt  | June 21, 2017 |  Posted In: Insight for DBAs, Insight for Developers, MongoDB PREVIOUS POST In this short blog post, we will discuss a helpful feature for tracing MongoDB queries: Cursor Comments. Cursor Comments Much like other database systems, MongoDB supports the ability for application developers to set comment strings on their database queries using the Cursor Comment feature. This feature is very useful for both DBAs and developers for quickly and efficiently tying a MongoDB query found on the database server to a line of code in the application source. Once ... Read More

Feed: Latest imported feed items on Analytics Matters. Making intelligent and accurate predictions is the core objective of machine learning and artificial intelligence applications. To achieve that objective, the machine learning or artificial intelligence application needs clean and well-organized information in a robust ecosystem architecture. Machine Learning (ML) is the process of a computer system making a prediction based on samples of past observations. There are various types of ML methods. One of the approaches is where the ML algorithm is trained using a labeled or unlabeled training data set to produce a model. New input data is introduced to the ... Read More

Feed: AWS Compute Blog. Author: Tara Van Unen.  Stephen Liedig, Solutions Architect One of the many challenges professional software architects and developers face is how to make cloud-native applications scalable, fault-tolerant, and highly available. Fundamental to your project success is understanding the importance of making systems highly cohesive and loosely coupled. That means considering the multi-dimensional facets of system coupling to support the distributed nature of the applications that you are building for the cloud. By that, I mean addressing not only the application-level coupling (managing incoming and outgoing dependencies), but also considering the impacts of of platform, spatial, and ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Erin McGill. By Erin McGill. Erin is a Partner Solutions Architect (SA) at AWS. Throughout this series, we’ve talked about easing cross-account role creation with AWS CloudFormation and a custom stack URL.  We’ve also discussed how to dynamically generate CloudFormation templates to populate a unique external ID. But once your cross-account role is created, how does the Amazon Resource Name (ARN) for the generated cross-account role in the customer account get back to your company’s onboarding portal? Asking the customer to cut and paste the output ARN is possible, but it’s a manual step ... Read More

Feed: All - O'Reilly Media. Author: Mike Loukides. Rhombus and hex weave. (source: Joel Cooper on Flickr). JupyterCon is August 22-25, 2017, in New York City. Registration is now open. What is Jupyter, and why do you care? After all, Jupyter has never become a buzzword like data science, artificial intelligence, or Web 2.0. Unlike those big abstractions, Jupyter is very concrete. It’s an open source project, a piece of software, that does specific things. But without attracting the hype, Jupyter Notebooks are revolutionizing the way engineers and data scientists work together. If all important work is collaborative, the most ... Read More