Tag: ACID

Feed: AWS Big Data Blog. Amazon EMR is a cloud big data platform for running large-scale distributed data processing jobs, interactive SQL queries, and machine learning (ML) applications using open-source analytics frameworks such as Apache Spark, Apache Hive, and Presto. Apache Iceberg is an open table format for huge analytic datasets. Table formats typically indicate the format and location of individual table files. Iceberg adds functionality on top of that to help manage petabyte-scale datasets as well as newer data lake requirements such as transactions, upsert/merge, time travel, and schema and partition evolution. Iceberg adds tables to compute engines including ... Read More

Feed: AWS Big Data Blog. Nowadays, many customers have built their data lakes as the core of their data analytic systems. In a typical use case of data lakes, many concurrent queries run to retrieve consistent snapshots of business insights by aggregating query results. A large volume of data constantly comes from different data sources into the data lakes. There is also a common demand to reflect the changes occurring in the data sources into the data lakes. This means that not only inserts but also updates and deletes need to be replicated into the data lakes. Apache Iceberg provides ... Read More

Feed: Recent Announcements. We are excited to announce the general availability of Amazon Athena ACID transactions, a new capability that adds insert, update, delete, and time travel operations to Athena's SQL data manipulation language (DML). Athena ACID transactions enable multiple concurrent users to make reliable, row-level modifications to their Amazon S3 data from Athena's console, API, and ODBC and JDBC drivers. Built on the Apache Iceberg table format, Athena ACID transactions are optimized for Amazon S3 storage, support seamless schema evolution, and ensure atomic operations across other services and engines that support the Iceberg table format such as Amazon EMR, ... Read More

Feed: Recent Announcements. We are excited to announce that Amazon EMR 6.5.0 now includes Apache Iceberg version 0.12. Apache Iceberg is an open table format for large data sets in Amazon S3 and provides fast query performance over large tables, atomic commits, concurrent writes, and SQL-compatible table evolution. With the current release, you can use Apache Spark 3.1.2 on EMR clusters with the Iceberg table format. Apache Iceberg offers an open source table format for data stored in data lakes that helps data engineers manage complex challenges such as managing continuously evolving data sets while maintaining query performance. Iceberg allows ... Read More

Feed: Recent Announcements. We are excited to announce the public preview of Amazon Athena ACID transactions, a new capability that adds write, delete, update, and time travel operations to Athena's SQL data manipulation language (DML). Athena ACID transactions enables multiple concurrent users to make reliable, row-level modifications to their Amazon S3 data from Athena's console, API, and ODBC and JDBC drivers. Built on the Apache Iceberg table format, Athena ACID transactions are compatible with other services and engines such as Amazon EMR and Apache Spark that support the Iceberg table format. Using Athena ACID transactions, you can now make business- ... Read More

Feed: AWS Big Data Blog. Data lakes on Amazon Simple Storage Service (Amazon S3) have become the default repository for all enterprise data and serve as common choice for a large number of users querying from a variety of analytics and ML tools. Often times you want to ingest data continuously into the data lake from multiple sources and query against the data lake from many analytics tools concurrently. Previously, getting consistent results meant that you had to build custom pipelines to keep data consistent and resulted in delays before the data became available to the consuming tools. We announced AWS Lake Formation transactions, row-level ... Read More

Feed: AWS Architecture Blog. In Part I of this two-part blog, we outlined best practices to consider when building resilient applications in hybrid on-premises/cloud environments. We also showed you how to adapt mindsets and organizational culture. In Part II, we’ll provide technical considerations related to architecture and patterns for resilience in AWS Cloud. Considerations on architecture and patterns The AWS Well-Architected Framework defines resilience as having “the capability to recover when stressed by load (more requests for service), attacks (either accidental through a bug, or deliberate through intention), and failure of any component in the workload’s components.” Resilience is an ... Read More

Feed: Planet PostgreSQL. While migrating to PostgreSQL, some of the users may be curious about the type of replication offered by PostgreSQL. Some might want to know if they have to pay any additional enterprise license in order to achieve this requirement. This is because, a few commercial databases require a customer to take an enteprise license to setup replication. But, PostgreSQL, like advanced Open Source databases, automatically offers several types of replication methods built-in. This means, we never have to consider taking any additional license or an enterprise solution and stay away from vendor lock-in while using PostgreSQL. In ... Read More

Feed: Matillion. Author: Rob Callaghan;  Unlock your lakehouse potential with Matillion’s cloud-native data integration platform With the arrival of Matillion ETL for Delta Lake on Databricks, data professionals across the organization can now leverage visually designed data transformations to build their lakehouse architecture and unify their data ecosystem.These days, companies need a place to store structured data within the cloud, and a cloud data warehouse fits that bill. But often in a modern data ecosystem, companies also have to deal with semi-structured data coming from a variety of data sources that doesn’t fit into that tabular data paradigm. Nor does it ... Read More

Feed: Actian. Author: Lewis Carr. In the opening installment of this blog series—Data Lakes, Data Warehouses and Data Hubs: Do we need another choice? I explore why simply migrating these on-prem data integration, management, and analytics platforms to the Cloud does not fully address modern data analytics needs. In comparing these three platforms, it becomes clear that all of them meet certain critical needs, but none of them meet the needs of business end-users without significant support from IT. In the second blog in this series —  What is a Data Analytics Hub? — I introduce the term Data Analytics ... Read More