Tag: ACID

Feed: Recent Announcements. We are excited to announce the general availability of Amazon Athena ACID transactions, a new capability that adds insert, update, delete, and time travel operations to Athena's SQL data manipulation language (DML). Athena ACID transactions enable multiple concurrent users to make reliable, row-level modifications to their Amazon S3 data from Athena's console, API, and ODBC and JDBC drivers. Built on the Apache Iceberg table format, Athena ACID transactions are optimized for Amazon S3 storage, support seamless schema evolution, and ensure atomic operations across other services and engines that support the Iceberg table format such as Amazon EMR, ... Read More

Feed: Recent Announcements. We are excited to announce that Amazon EMR 6.5.0 now includes Apache Iceberg version 0.12. Apache Iceberg is an open table format for large data sets in Amazon S3 and provides fast query performance over large tables, atomic commits, concurrent writes, and SQL-compatible table evolution. With the current release, you can use Apache Spark 3.1.2 on EMR clusters with the Iceberg table format. Apache Iceberg offers an open source table format for data stored in data lakes that helps data engineers manage complex challenges such as managing continuously evolving data sets while maintaining query performance. Iceberg allows ... Read More

Feed: Recent Announcements. We are excited to announce the public preview of Amazon Athena ACID transactions, a new capability that adds write, delete, update, and time travel operations to Athena's SQL data manipulation language (DML). Athena ACID transactions enables multiple concurrent users to make reliable, row-level modifications to their Amazon S3 data from Athena's console, API, and ODBC and JDBC drivers. Built on the Apache Iceberg table format, Athena ACID transactions are compatible with other services and engines such as Amazon EMR and Apache Spark that support the Iceberg table format. Using Athena ACID transactions, you can now make business- ... Read More

Feed: AWS Big Data Blog. Data lakes on Amazon Simple Storage Service (Amazon S3) have become the default repository for all enterprise data and serve as common choice for a large number of users querying from a variety of analytics and ML tools. Often times you want to ingest data continuously into the data lake from multiple sources and query against the data lake from many analytics tools concurrently. Previously, getting consistent results meant that you had to build custom pipelines to keep data consistent and resulted in delays before the data became available to the consuming tools. We announced AWS Lake Formation transactions, row-level ... Read More

Feed: AWS Architecture Blog. In Part I of this two-part blog, we outlined best practices to consider when building resilient applications in hybrid on-premises/cloud environments. We also showed you how to adapt mindsets and organizational culture. In Part II, we’ll provide technical considerations related to architecture and patterns for resilience in AWS Cloud. Considerations on architecture and patterns The AWS Well-Architected Framework defines resilience as having “the capability to recover when stressed by load (more requests for service), attacks (either accidental through a bug, or deliberate through intention), and failure of any component in the workload’s components.” Resilience is an ... Read More

Feed: Planet PostgreSQL. While migrating to PostgreSQL, some of the users may be curious about the type of replication offered by PostgreSQL. Some might want to know if they have to pay any additional enterprise license in order to achieve this requirement. This is because, a few commercial databases require a customer to take an enteprise license to setup replication. But, PostgreSQL, like advanced Open Source databases, automatically offers several types of replication methods built-in. This means, we never have to consider taking any additional license or an enterprise solution and stay away from vendor lock-in while using PostgreSQL. In ... Read More

Feed: Matillion. Author: Rob Callaghan;  Unlock your lakehouse potential with Matillion’s cloud-native data integration platform With the arrival of Matillion ETL for Delta Lake on Databricks, data professionals across the organization can now leverage visually designed data transformations to build their lakehouse architecture and unify their data ecosystem.These days, companies need a place to store structured data within the cloud, and a cloud data warehouse fits that bill. But often in a modern data ecosystem, companies also have to deal with semi-structured data coming from a variety of data sources that doesn’t fit into that tabular data paradigm. Nor does it ... Read More

Feed: Actian. Author: Lewis Carr. In the opening installment of this blog series—Data Lakes, Data Warehouses and Data Hubs: Do we need another choice? I explore why simply migrating these on-prem data integration, management, and analytics platforms to the Cloud does not fully address modern data analytics needs. In comparing these three platforms, it becomes clear that all of them meet certain critical needs, but none of them meet the needs of business end-users without significant support from IT. In the second blog in this series —  What is a Data Analytics Hub? — I introduce the term Data Analytics ... Read More

Feed: Matillion. Author: Julie Polito; 2020 was a year of rapid and tumultuous change for nearly every business – especially in the cloud. Our need to work 100 percent remotely, and our need to have data accessible to us no matter where we worked, accelerated the move to the cloud for many organizations. We’re likely to see that trend continuing into 2021 as many organizations decide that the remote way of work and sharing information via virtual channels is part of the new normal. But along with organizations speeding up a move to the cloud, we’re seeing another long awaited macro ... Read More

Feed: AWS Big Data Blog. We announced the preview of AWS Lake Formation transactions, row-level security, and acceleration at AWS re:Invent 2020. In Part 1 of this series, we explained how to set up a governed table and add objects to it. In this post, we expand on this example, and demonstrate how to ingest streaming data into governed tables using Lake Formation transactions. In typical streaming use cases, new data is continuously ingested into a data lake. This creates many small files that may impact query performance. In addition, a common requirement is to isolate queries from updates without requiring downtime or having ... Read More