Category: Datawarehousing

Feed: Cloud Data Integration Software | Matillion. Author: Laura Malins, Solution Architect; Laura Malins, Solution ArchitectIn this blog, we will show you how to use grid variable to apply business rules in a Transformation job within Matillion ETL. It is often the case that business rules are hard-coded into an ETL meaning any change to the business rules requires an ETL developer to update the logic. Here we look at an alternative approach.Using Grid Variables Apply Business Rules with Matillion ETLIn this example we look at product transactions where the product code and region it is sold will affect the amount ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Ian Funnell, Lead Solution Architect; Ian Funnell, Lead Solution ArchitectThis blog describes how to build a single, generic method for data transformation (the T of ELT). You would use it to apply the newly-loaded changed data into its permanent home in a replicated table in the data warehouse.One of the first things you’ll want to do with Matillion is copy data out from source systems into the target data warehouse. Every source system usually contains multiple tables, each one having a different set of columns.Every source system has its own mechanism for ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Craig Rouse, Solution Architect; Craig Rouse, Solution ArchitectMatillion uses the Extract-Load-Transform (ELT) approach to delivering quick results for a wide range of data processing purposes: everything from customer behavior analytics, financial analysis, and even reducing the cost of synthesizing DNA.The Zendesk Query component in Matillion ETL for BigQuery presents an easy-to-use graphical interface, enabling you to easily pull data from Zendesk directly into BigQuery. Many of our customers use the Zendesk Query to automate and enhance their customer support with Zendesk Support.The connector is completely self-contained: no additional software installation is required ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Ed Thompson, CTO; The latest release of Matillion ETL for Amazon Redshift version 1.35, includes a whole host of enhancements, new features and new components to make Matillion better than ever before!What’s new in Matillion ETL for Amazon Redshift version 1.35?You can also find the full release notes and upgrade instructions on our Amazon Redshift Support Center.New Data Connector ComponentsNew/Updated ComponentsNew “Data Transfer” Component that boasts all the functionality of the existing S3 Get, S3 Put, and Cloud Storage Put components, plus additional source and target destinations (Azure Blob Storage).‘Extract To New Job’ function available by ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Ed Thompson, CTO; The latest release of Matillion ETL for BigQuery version 1.35, includes a whole host of enhancements, new features and new components to make Matillion better than ever before!What’s new in Matillion ETL for BigQuery version 1.35?You can also find the full release notes and upgrade instructions on our BigQuery Support Center.New Data Connector ComponentsNew/Updated Components “Data Transfer” Component that boasts all the functionality of the existing S3 Get, S3 Put, and Cloud Storage Put components, plus additional source and target destinations (Azure Blob Storage).New ‘Extract To New Job’ function available by right-clicking a selection of ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Ed Thompson, CTO; The latest release of Matillion ETL for Snowflake version 1.35, includes a whole host of enhancements, new features and new components to make Matillion ETL better than ever before!What’s new in Matillion ETL for Snowflake version 1.35?You can also find the full release notes and upgrade instructions on our Snowflake Support Center.New Data Connector ComponentNew/Updated Components and FeaturesNew “Data Transfer” Component that boasts all the functionality of the existing S3 Get, S3 Put, and Cloud Storage Put components, plus additional source and target destinations (Azure Blob Storage).Support for running on Azure platformSupport for Snowflake’s AUTOINCREMENT option in the ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Guest Author; Editor note: Today’s blog post was prepared by work placement students from Saint Ambrose school in Altrincham, UK.This blog will walk you through the configuration process for setting up an ‘OK Google’ command to run your Matillion ETL job from a Google Home smart device or ‘OK Google’ enabled mobile phone.To connect the Google Home device with Matillion an API call needs to be made from Google Home which puts a message on an SQS Queue to be picked up by Matillion. The workflow is below. This blog will focus ... Read More

Feed: Cloud Data Integration Software | Matillion. Author: Arawan Gajajiva, Solution Architect; Arawan Gajajiva, Solution ArchitectMatillion uses the Extract-Load-Transform (ELT) approach to delivering quick results for a wide range of data processing purposes: everything from customer behavior analytics, financial analysis, and even reducing the cost of synthesizing DNA.The Open Exchange Rates Query component presents an easy-to-use graphical interface, enabling you to pull data from the Open Exchange Rates API directly into Amazon Redshift. This component is very popular with our customers to source live and historical currency exchange rate data, allowing for currency conversions of data loaded into Amazon Redshift ... Read More

Feed: Actian. Author: Pradeep Bhanot. Over the past two decades data warehouse solutions have evolved and diverged to address a myriad of use cases. Meanwhile, the pace of business continues to accelerate, making it harder to remain competitive. These new demands can stretch the abilities of traditional data warehouses. Below are 5 of the common pitfalls that can trip up traditional data warehouses.Currency: As the demands for organizations to operate in real-time or in the moment increase, data warehouses need to deliver ever more current data. SQL Hadoop databases commonly fail to handle continuous streams of updates as the file ... Read More

Feed: MemSQL Blog. Author: Floyd Smith. Apache Kafka is widely used to transmit and store messages between applications. Kafka is fast and scalable, like MemSQL. MemSQL and Kafka work well together. You can find out more, and see a demo, in our recent webinar. In the webinar, Senior Sales Engineer Neil Dahlke of MemSQL explains how Apache Kafka and MemSQL can be used together to help reduce latency. Latency has three main causes: Slow data loading – due to database single-node architectures and locking mechanisms, it can take many minutes – even hours – to load datasets into the system ... Read More