Tag: Data Lake

Feed: Alation. Author: Jason Rushin. October 21, 2021 — Data Swamp vs Data Lake When you imagine a lake, it’s likely an idyllic image of a tree-ringed body of reflective water amid singing birds and dabbling ducks. But a swamp, on the other hand, is dark and dank, full of scary creatures, heavy wet air, and either a poisonous frog or an angry alligator behind every dead tree snag. I’ll take the lake, thank you very much. And so will your data. You know the story well: you have a ton of data and need fast access to the right ... Read More

Feed: AWS Partner Network (APN) Blog. Author: Jitesh Bhattacharjee. By Jitesh Bhattacharjee, Delivery Partner – TCSBy Nicolas Weydert, Chief Architect – TCSBy Sanjay Gupta, Sr. Partner Solutions Architect – AWS TCS Many organizations leverage unstructured data collected from social media feeds, stock streaming, and data clickstream to gain insights about the needs of their customers. They use this information to customize their products and improve customer experience using data lake solutions. A Lake House architecture is defined by a central repository (data lake) which allows ingestion of unstructured, structured, and real-time data that’s consumed by various processes like analytics engine, ... Read More

Feed: AWS Big Data Blog. This post is co-written with Mat Sharpe, Technical Lead, AWS & Systems Engineering from Integral Ad Science. Integral Ad Science (IAS) is a global leader in digital media quality. The company’s mission is to be the global benchmark for trust and transparency in digital media quality for the world’s leading brands, publishers, and platforms. IAS does this through data-driven technologies with actionable real-time signals and insight. In this post, we discuss how IAS uses AWS Lake Formation and Amazon Athena to efficiently manage governance and security of data. The challenge IAS processes over 100 billion ... Read More

Feed: SAS Blogs. Author: Jim Harris. Know the difference between data lake and data warehouse? Read this article. What's different with business technology today compared to 10 years ago? It might be better to ask what has not changed. Think about huge leaps in the capabilities of data and analytics technologies and platforms – like cloud computing and big data cloud – along with easier data access, remarkable differences in service delivery, and a growing need for enterprise-wide collaboration. To make the most of new business opportunities accompanying these trends, many organizations have turned to the data lake. But before ... Read More

Feed: AWS Big Data Blog. This is a guest blog post co-written by Leonardo Pêpe, a Data Engineer at MOIA. MOIA is an independent company of the Volkswagen Group with locations in Berlin and Hamburg, and operates its own ride pooling services in Hamburg and Hanover. The company was founded in 2016 and develops mobility services independently or in partnership with cities and existing transport systems. MOIA’s focus is on ride pooling and the holistic development of the software and hardware for it. In October 2017, MOIA started a pilot project in Hanover to test a ride pooling service, which ... Read More

Feed: AWS Big Data Blog. If you use data lakes in Amazon Simple Storage Service (Amazon S3) and use SAP HANA as your transactional data store, you may need to join the data in your data lake with SAP HANA in the cloud, SAP HANA running on Amazon Elastic Compute Cloud (Amazon EC2), or with an on-premises SAP HANA, for example to build a dashboard or create consolidated reporting. In such use cases, Amazon Athena Federated Query allows you to seamlessly access the data from SAP HANA database without building ETL pipelines to copy or unload the data to the S3 data ... Read More

Feed: AWS Big Data Blog. Thousands of customers are building petabyte-scale data lakes on AWS. Many of these customers use AWS Lake Formation to easily build and share their data lakes across the organization. As the number of tables and users increase, data stewards and administrators are looking for ways to manage permissions on data lakes easily at scale. Customers are struggling with “role explosion” and need to manage hundreds or even thousands of user permissions to control data access. For example, for an account with 1,000 resources and 100 principals, the data steward would have to create and manage ... Read More

Feed: AWS Big Data Blog. On July 16, 2021, Amazon Athena upgraded its Apache Hudi integration with new features and support for Hudi’s latest 0.8.0 release. Hudi is an open-source storage management framework that provides incremental data processing primitives for Hadoop-compatible data lakes. This upgraded integration adds the latest community improvements to Hudi along with important new features including snapshot queries, which provide near real-time views of table data, and reading bootstrapped tables which provide efficient migration of existing table data. In this series of posts on Athena and Hudi, we will provide a short overview of key Hudi capabilities ... Read More

Feed: AWS Big Data Blog. If you use data lakes in Amazon Simple Storage Service (Amazon S3) and use Teradata as your transactional data store, you may need to join the data in your data lake with Teradata in the cloud, Teradata running on Amazon Elastic Compute Cloud (Amazon EC2), or with an on-premises Teradata database, for example to build a dashboard or create consolidated reporting. In these use cases, the Amazon Athena Federated Query feature allows you to seamlessly access the data from Teradata database without having to move the data to your S3 data lake. This removes the ... Read More

Feed: AWS Big Data Blog. If you use data lakes in Amazon Simple Storage Service (Amazon S3) and use Snowflake as your data warehouse solution, you may need to join your data in your data lake with Snowflake. For example, you may want to build a dashboard by joining historical data in your Amazon S3 data lake and the latest data in your Snowflake data warehouse or create consolidated reporting. In such use cases, Amazon Athena Federated Query allows you to seamlessly access the data from Snowflake without building ETL pipelines to copy or unload the data to the S3 ... Read More