Tag: Spark

Feed: Recent Announcements. Amazon SageMaker announces a new set of capabilities that will enable interactive Spark based data processing from SageMaker Studio Notebooks. Amazon SageMaker Studio is the first fully integrated development environment (IDE) for machine learning (ML). SageMaker Studio provides a single, web-based visual interface where you can perform all ML development steps required to prepare data, as well as build, train, and deploy models. With a single click, data scientists and developers can quickly spin up Studio Notebooks to interactively explore datasets and build ML models. Starting today, data scientists and data engineers can visually browse, discover, and ... Read More

Feed: AWS Big Data Blog. Amazon EMR on Amazon EKS is a deployment option for Amazon EMR that allows you to run Apache Spark on Amazon Elastic Kubernetes Service (Amazon EKS). If you run open-source Apache Spark on Amazon EKS, you can now use Amazon EMR to automate provisioning and management, and run Apache Spark up to three times faster. If you already use Amazon EMR, you can run Amazon EMR-based Apache Spark applications with other types of applications on the same Amazon EKS cluster to improve resource utilization and simplify infrastructure management. Earlier this year, we launched support for ... Read More

Feed: AWS Big Data Blog. Customers today want to focus more on their core business model and less on the underlying infrastructure and operational burden. As customers migrate to the AWS Cloud, they’re realizing the benefits of being able to innovate faster on their own applications by relying on AWS to handle big data platforms, operations, and automation. Many of AWS’s customers have migrated their big data workloads from on premises to Amazon Elastic Compute Cloud (Amazon EC2) and Amazon EMR, and process large amounts of data to get insights from it in a secure and cost-effective manner. If you’re ... Read More

Feed: AWS Big Data Blog. Increasingly, a business’s success depends on its agility in transforming data into actionable insights, which requires efficient and automated data processes. In the previous post – Build a SQL-based ETL pipeline with Apache Spark on Amazon EKS, we described a common productivity issue in a modern data architecture. To address the challenge, we demonstrated how to utilize a declarative approach as the key enabler to improve efficiency, which resulted in a faster time to value for businesses. Generally speaking, managing applications declaratively in Kubernetes is a widely adopted best practice. You can use the same ... Read More

Feed: Recent Announcements. Today, we’re pleased to announce AWS Glue version 3.0, a new version of AWS Glue Spark for your batch and streaming jobs that accelerates your data integration workloads in AWS. AWS Glue 3.0 introduces a performance-optimized Spark runtime that includes optimizations from AWS Glue and Amazon EMR, and is based on open-source Apache Spark 3.1.1. The AWS Glue 3.0 runtime optimizes both read and write access to Amazon Simple Storage Service (Amazon S3), using faster vectorized readers and Amazon S3 optimized output committers. It also optimizes access to the AWS Glue Data Catalog with the use of partition ... Read More

Feed: AWS Big Data Blog. In August 2020, we announced the availability of AWS Glue 2.0. AWS Glue 2.0 reduced job startup times by 10x, enabling customers to reali­­ze an average of 45% cost savings on their extract, transform, and load (ETL) jobs. The fast start time allows customers to easily adopt AWS Glue for batching, micro-batching, and streaming use cases. In the last year, AWS Glue has evolved from an ETL service to a serverless data integration service, offering all the required capabilities needed to build, operate and scale a modern data platform. The following are some of the ... Read More

Feed: Recent Announcements. You can now set up SQL-Based ETL with Apache Spark on Amazon EKS. This solutions implementation provides declarative data processing support, codeless extract-transform-load (ETL) capabilities, and workflow orchestration automation to help data scientists and analysts access their data and create meaningful insights without the need for manual IT processes. This solution abstracts common ETL activities, including formatting, partitioning, and transforming datasets, into configurable and productive data processes. This abstraction results in actionable insights that is derived quickly to help you accelerate your data-driven business decisions. Additionally, the solution uses an open-source Arc data processing framework and powered ... Read More

Feed: AWS Big Data Blog. Organizations often need to process large volumes of data before serving to business stakeholders. In this blog, we will learn how to leverage Amazon EMR to process data using Apache Spark, the go-to platform for in-memory analytics of large data volume, and connect business intelligence (BI) tool Amazon QuickSight to serve data to end-users. QuickSight is a fast, cloud-powered BI service that makes it easy to build visualizations, perform ad hoc analysis, and quickly get business insights from your data. With our cloud-based service, you can easily connect to your data, perform advanced analysis, and ... Read More

Feed: AWS Big Data Blog. Today, the most successful and fastest growing companies are generally data-driven organizations. Taking advantage of data is pivotal to answering many pressing business problems; however, this can prove to be overwhelming and difficult to manage due to data’s increasing diversity, scale, and complexity. One of the most popular technologies that businesses use to overcome these challenges and harness the power of their growing data is Apache Spark. Apache Spark is an open-source, distributed data processing framework capable of performing analytics on large-scale datasets, enabling businesses to derive insights from all of their data whether it ... Read More

Feed: Microsoft Azure Blog. Author: Daniel Coelho. This blog post has been co-authored by Bhanu Prakash, Principal Program Manager, Azure Databricks. We are now announcing the general availability of the Apache Spark 3.0 compatible Apache Spark Connector for SQL Server and Azure SQL, accessible through Maven. The Spark 3.0 compatible connector went into preview early this year. Since then, we have seen tremendous customer adoption and received helpful customer feedback. Over the last few months, after incorporating enhancements and bug fixes to the connector, we are now excited about the general availability of this connector so that customers can expand their usage ... Read More