Tag: ACID

Feed: Recent Announcements. We are excited to announce the public preview of Amazon Athena ACID transactions, a new capability that adds write, delete, update, and time travel operations to Athena's SQL data manipulation language (DML). Athena ACID transactions enables multiple concurrent users to make reliable, row-level modifications to their Amazon S3 data from Athena's console, API, and ODBC and JDBC drivers. Built on the Apache Iceberg table format, Athena ACID transactions are compatible with other services and engines such as Amazon EMR and Apache Spark that support the Iceberg table format. Using Athena ACID transactions, you can now make business- ... Read More

Feed: AWS Big Data Blog. Data lakes on Amazon Simple Storage Service (Amazon S3) have become the default repository for all enterprise data and serve as common choice for a large number of users querying from a variety of analytics and ML tools. Often times you want to ingest data continuously into the data lake from multiple sources and query against the data lake from many analytics tools concurrently. Previously, getting consistent results meant that you had to build custom pipelines to keep data consistent and resulted in delays before the data became available to the consuming tools. We announced AWS Lake Formation transactions, row-level ... Read More

Feed: AWS Architecture Blog. In Part I of this two-part blog, we outlined best practices to consider when building resilient applications in hybrid on-premises/cloud environments. We also showed you how to adapt mindsets and organizational culture. In Part II, we’ll provide technical considerations related to architecture and patterns for resilience in AWS Cloud. Considerations on architecture and patterns The AWS Well-Architected Framework defines resilience as having “the capability to recover when stressed by load (more requests for service), attacks (either accidental through a bug, or deliberate through intention), and failure of any component in the workload’s components.” Resilience is an ... Read More

Feed: Planet PostgreSQL. While migrating to PostgreSQL, some of the users may be curious about the type of replication offered by PostgreSQL. Some might want to know if they have to pay any additional enterprise license in order to achieve this requirement. This is because, a few commercial databases require a customer to take an enteprise license to setup replication. But, PostgreSQL, like advanced Open Source databases, automatically offers several types of replication methods built-in. This means, we never have to consider taking any additional license or an enterprise solution and stay away from vendor lock-in while using PostgreSQL. In ... Read More

Feed: Matillion. Author: Rob Callaghan;  Unlock your lakehouse potential with Matillion’s cloud-native data integration platform With the arrival of Matillion ETL for Delta Lake on Databricks, data professionals across the organization can now leverage visually designed data transformations to build their lakehouse architecture and unify their data ecosystem.These days, companies need a place to store structured data within the cloud, and a cloud data warehouse fits that bill. But often in a modern data ecosystem, companies also have to deal with semi-structured data coming from a variety of data sources that doesn’t fit into that tabular data paradigm. Nor does it ... Read More

Feed: Actian. Author: Lewis Carr. In the opening installment of this blog series—Data Lakes, Data Warehouses and Data Hubs: Do we need another choice? I explore why simply migrating these on-prem data integration, management, and analytics platforms to the Cloud does not fully address modern data analytics needs. In comparing these three platforms, it becomes clear that all of them meet certain critical needs, but none of them meet the needs of business end-users without significant support from IT. In the second blog in this series —  What is a Data Analytics Hub? — I introduce the term Data Analytics ... Read More

Feed: Matillion. Author: Julie Polito; 2020 was a year of rapid and tumultuous change for nearly every business – especially in the cloud. Our need to work 100 percent remotely, and our need to have data accessible to us no matter where we worked, accelerated the move to the cloud for many organizations. We’re likely to see that trend continuing into 2021 as many organizations decide that the remote way of work and sharing information via virtual channels is part of the new normal. But along with organizations speeding up a move to the cloud, we’re seeing another long awaited macro ... Read More

Feed: AWS Big Data Blog. We announced the preview of AWS Lake Formation transactions, row-level security, and acceleration at AWS re:Invent 2020. In Part 1 of this series, we explained how to set up a governed table and add objects to it. In this post, we expand on this example, and demonstrate how to ingest streaming data into governed tables using Lake Formation transactions. In typical streaming use cases, new data is continuously ingested into a data lake. This creates many small files that may impact query performance. In addition, a common requirement is to isolate queries from updates without requiring downtime or having ... Read More

Feed: Planet MySQL; Author: Michael McLaughlin; The idea of ACID transactions are a basic feature of SQL’s individual Data Manipulation Language (DML) commands, like the INSERT, UPDATE, and DELETE statements. Transactions across two or more tables are a natural extension of ACID compliance features provided by DML commands. However, they require a structured programming approach, like a store procedure or like API implemented in an imperative language. Surprisingly, transaction management wasn’t covered well in Alan Beaulieu’s Learning SQL because he only provided pseudo code logic. While I thought troubleshoot some broken MySQL SQL/PSM logic would be a good learning experience ... Read More

Feed: AWS Big Data Blog. Thousands of customers are building their data lakes on Amazon Simple Storage Service (Amazon S3). You can use AWS Lake Formation to build your data lakes easily—in a matter of days as opposed to months. However, there are still some difficult challenges to address with your data lakes: Supporting streaming updates and deletes in your data lakes, for example, database replication, and supporting privacy regulations such as GDPR and CCPA Achieving fine-grained secure sharing not only with table-level or column-level access control, but with row-level access control Optimizing the layout of various tables and files on ... Read More