Tag: Apache

Feed: AWS Big Data Blog. Amazon EMR on Amazon EKS is a deployment option for Amazon EMR that allows you to run Apache Spark on Amazon Elastic Kubernetes Service (Amazon EKS). If you run open-source Apache Spark on Amazon EKS, you can now use Amazon EMR to automate provisioning and management, and run Apache Spark up to three times faster. If you already use Amazon EMR, you can run Amazon EMR-based Apache Spark applications with other types of applications on the same Amazon EKS cluster to improve resource utilization and simplify infrastructure management. Earlier this year, we launched support for ... Read More

Feed: AWS Architecture Blog. Using analytics to gain insights from a variety of datasets is key to successful transformation. There are many options to consider to realize the full value and potential of our data in a hybrid cloud infrastructure. Common practice is to route data produced from on-premises to a central repository or data lake. Here it can be consumed by multiple applications. You can use an Apache Kafka cluster for data movement from on-premises to the data lake, using Amazon Simple Storage Service (Amazon S3). But you must either replicate the topics onto a cloud cluster, or develop ... Read More

Feed: Planet MySQL; Author: Frederic Descamps; We already saw how easy it’s to deploy solutions on OCI using Terraform and Resource Manager’s Stack. I’ve published several resources available on this page. Today we will see how easy it’s to deploy Apache Superset on OCI using MySQL Data Service. Apache Superset is an open source BI, Reporting, Charting tool that competes with Tableau, Looker, etc.  For a list of companies that have deployed Superset, see:  https://github.com/apache/superset/blob/master/RESOURCES/INTHEWILD.md. Superset is loaded with options that make it easy for users of all skill sets to explore and visualize their data, from simple line charts to ... Read More

Feed: Recent Announcements. Amazon Managed Workflows for Apache Airflow (MWAA) is now available in six new AWS Regions: South America (Sao Paulo), Asia Pacific (Seoul), Asia Pacific (Mumbai), Canada (Central), EU (London), and EU (Paris). Amazon MWAA is a managed service for Apache Airflow that lets you use the same familiar Apache Airflow platform as you do today to orchestrate your workflows and enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure. Learn more about using Amazon MWAA on the product page. Amazon MWAA is now available in US (Northern Virginia), US ... Read More

Feed: AWS Big Data Blog. Customers today want to focus more on their core business model and less on the underlying infrastructure and operational burden. As customers migrate to the AWS Cloud, they’re realizing the benefits of being able to innovate faster on their own applications by relying on AWS to handle big data platforms, operations, and automation. Many of AWS’s customers have migrated their big data workloads from on premises to Amazon Elastic Compute Cloud (Amazon EC2) and Amazon EMR, and process large amounts of data to get insights from it in a secure and cost-effective manner. If you’re ... Read More

Feed: AWS Big Data Blog. In August 2020, we announced the availability of AWS Glue 2.0. AWS Glue 2.0 reduced job startup times by 10x, enabling customers to reali­­ze an average of 45% cost savings on their extract, transform, and load (ETL) jobs. The fast start time allows customers to easily adopt AWS Glue for batching, micro-batching, and streaming use cases. In the last year, AWS Glue has evolved from an ETL service to a serverless data integration service, offering all the required capabilities needed to build, operate and scale a modern data platform. The following are some of the ... Read More

Feed: AWS Architecture Blog. Cloud Architects developing autonomous driving and ADAS workflows are challenged by loosely distributed process steps along the tool chain in hybrid environments. This is accelerated by the need to create a holistic view of all running pipelines and jobs. Common challenges include: finding and getting access to the data sources specific to your use case, understanding the data, moving and storing it, cleaning and transforming it and, preparing it for downstream consumption. Verifying that your data pipeline works correctly and ensuring you provide a certain data quality while your application is being developed adds to the ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Paul Scott-Murphy. By Paul Scott-Murphy, Chief Technology Officer – WANdiscoBy Roy Hasson, Principal Product Manager – AWS Glue / AWS Lake Formation WANdisco is the LiveData company and an AWS ISV Partner with the Migration Competency that provides technology and software products that are used to simplify and automate the migration of big data to the cloud. In this post, we’ll explain the challenges of migrating large, complex, actively-used structured datasets to Amazon Web Service (AWS), and how the combination of WANdisco LiveData Migrator, Amazon Simple Storage Service (Amazon S3), and AWS Glue ... Read More

Feed: Recent Announcements. You can now set up SQL-Based ETL with Apache Spark on Amazon EKS. This solutions implementation provides declarative data processing support, codeless extract-transform-load (ETL) capabilities, and workflow orchestration automation to help data scientists and analysts access their data and create meaningful insights without the need for manual IT processes. This solution abstracts common ETL activities, including formatting, partitioning, and transforming datasets, into configurable and productive data processes. This abstraction results in actionable insights that is derived quickly to help you accelerate your data-driven business decisions. Additionally, the solution uses an open-source Arc data processing framework and powered ... Read More

Feed: AWS Big Data Blog. On July 16, 2021, Amazon Athena upgraded its Apache Hudi integration with new features and support for Hudi’s latest 0.8.0 release. Hudi is an open-source storage management framework that provides incremental data processing primitives for Hadoop-compatible data lakes. This upgraded integration adds the latest community improvements to Hudi along with important new features including snapshot queries, which provide near real-time views of table data, and reading bootstrapped tables which provide efficient migration of existing table data. In this series of posts on Athena and Hudi, we will provide a short overview of key Hudi capabilities ... Read More