Tag: Spark

Feed: AWS Big Data Blog. Apache Spark is a powerful data processing engine that gives data analyst and engineering teams easy to use APIs and tools to analyze their data, but it can be challenging for teams to manage their Python and R library dependencies. Installing every dependency that a job may need before it runs and dealing with library version conflicts is time-consuming and complicated. Amazon EMR 6.0.0 simplifies this by allowing you to use Docker images from Docker Hub and Amazon ECR to package your dependencies. This allows you to package and manage your dependencies for individual Spark ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Scott Kellish. By Scott Kellish, Partner Solutions Architects at AWSBy Basil Faruqui, Principal Solutions Marketing Manager at BMC Digital Business AutomationBy Joe Goldberg, Innovation Evangelist at BMCPredictive maintenance, which analyzes sensor data to predict equipment failures, has emerged as one of the most common business use cases of machine learning (ML) and the Internet of Things (IoT). To build and train ML models, you need data science expertise. If you’re going to run those ML models in production at scale, you need data engineering expertise to build a pipeline for data ingestion, storage, ... Read More

Feed: Recent Announcements. AWS Glue now supports the ability to update the schema in the Glue Data Catalog from Glue Spark ETL jobs. Previously, you had to run Glue crawlers to create new tables or add new partitions to existing tables after running your Glue Spark ETL jobs resulting in additional cost and time ... Read More

Feed: Recent Announcements. Starting today, you can use Glue Spark ETL jobs to read, transform, and load data from Amazon DocumentDB (with MongoDB compatibility) and MongoDB tables into services such as Amazon S3 and Amazon Redshift for downstream analytics ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Sanjay Nighojkar. By Suraj Nathani, Sr. Solutions Architect at Mactores CognitionBy Anil Patel, Data Engineer at Mactores CognitionBy Sanjay Nighojkar, Manager, Partner Solutions Architect at AWSSeagate Technology is a United States-based data storage company with worldwide manufacturing facilities that generate huge amounts of manufacturing and testing data. With petabytes of data accumulated over 20 years, and more being generated each day, it was imperative for Seagate to have systems in place to ensure the cost of collecting, storing, and processing data did not exceed their return on investment (ROI). Seagate’s on-premises big data ... Read More

Feed: Matillion. Author: Julie Polito; Over the years, I’ve been extremely fortunate to be a part of some truly amazing teams. Thinking back, if I had to choose just one attribute that contributed to our shared success, it would be the relentless pursuit of greater efficiency. This is, of course, a very broad and multi-faceted topic where  opportunities abound, so it can be difficult to decide where to focus time and energy when expounding upon it. But in this case, it’s pretty clear what made the difference. No surprise, It’s collaboration It’s no coincidence that these particular teams placed a ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: William Vorhies. Summary:  An interesting documentary about the earliest days of AI/ML and my alternate take on how we should really be describing the development of our profession to the newly initiated. What could be more natural in these days of the Corona Virus than binge watching great videos.  I was intrigued by a recent offer from Futurology who has sponsored “The History of Artificial Intelligence” on YouTube for free viewing.  The two-and-a-half-hour video is described as "the culmination of documentaries that cover the history and origins of computing-based artificial intelligence." ... Read More

Feed: R-bloggers. Author: Random R Ramblings. [This article was first published on Random R Ramblings, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The first official release of poorman (v 0.1.9) is now on CRAN! You can now install poorman directly from CRAN with the following code: install.packages("poorman") In this blog post I want to address some common questions that I have received since I started writing the package. poorman is a package ... Read More

Feed: Alation. Author: Dave Wells. March 31, 2020— As organizations become more data driven and the scope of data consumers expands through self-service, data literacy and data fluency have become valuable organizational competencies. Current events throughout the world are escalating the value of these competencies from valuable to imperative. The COVID-19 pandemic affects all dimensions of business and society. As the pandemic moves from current events to recent history, data competencies will be essential for businesses to manage the economic, social, and regulatory consequences. What is Data Literacy? Data literacy is the ability to derive meaningful information from data. Literacy ... Read More

Author: . For decades organizations have relied on monolithic databases to run core business operations, but these traditional relational database management systems weren’t designed to support the requirements of modern application architectures.Modern tools and technologies allow developers to design and build web-scale microservices and container-based applications, and non-scalable centralised databases just don’t work in this type of architecture. In fact, they become a real limitation, particularly for mission-critical applications moving to distributed or cloud environments. Recently I presented a webinar on how enterprises can move beyond these limitations with a distributed SQL database, and I got some great questions, so ... Read More