Tag: column

Feed: R-bloggers. Author: hrbrmstr. I’ve blogged about how to use Amazon Athena with R before and if you are a regular Athena user, you’ve likely run into a situation where you prepare a dplyr chain, fire off a collect() and then wait. And, wait. And, wait. And, wait. Queries that take significant processing time or have large result sets do not play nicely with the provided ODBC and JDBC drivers. This means “hung” R sessions and severe frustration, especially when you can login to the AWS Athena console and see that the results are right there!! I’ve been crafting SQL ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Saumitra Buragohain. We’re excited to announce the long-awaited release of Hortonworks Data Platform 3.0.0. HDP 3.0 is faster, smarter, hybrid, bigger, trusted and real-time database. We encourage you to read our blog that announced HDP 3.0. You can also view our keynote presentation from DataWorks Summit, where we demoed an autonomous car (1/10 scale), trained with HDP 3.0 technologies. We also want to thank  our current and prospective customers and partners for signing up for the HDP 3.0 Early Access and we are encouraged by the huge interest- now, you can use the HDP ... Read More

Feed: AWS Big Data Blog. Exploratory data analysis of genomic datasets using ADAM and Mango with Apache Spark on Amazon EMRAs the cost of genomic sequencing has rapidly decreased, the amount of publicly available genomic data has soared over the past couple of years. New cohorts and studies have produced massive datasets consisting of over 100,000 individuals. Simultaneously, these datasets have been processed to extract genetic variation across populations, producing mass amounts of variation data for each cohort. In this era of big data, tools like Apache Spark have provided a user-friendly platform for batch processing of large datasets. However, ... Read More

Feed: Planet MySQL; Author: Severalnines; Before you attempt to perform any schema changes on your production databases, you should make sure that you have a rock solid rollback plan; and that your change procedure has been successfully tested and validated in a separate environment. At the same time, it’s your responsibility to make sure that the change causes none or the least possible impact acceptable to the business. It’s definitely not an easy task. In this article, we will take a look at how to perform database changes on MySQL and MariaDB in a controlled way. We will talk about ... Read More

Feed: MemSQL Blog. Author: Rick Negrin. It’s time for us to admit what we have all known is true for a long time; NoSQL is the wrong tool for many of the modern application use cases, and it’s time that we move on. NoSQL came into existence because the databases at the time couldn’t handle the scale required. The rise of this new generation of data services solved many of the problems of web scale and rapidly growing data sets when it was created more than a decade ago. NoSQL also offered a new, cost-effective method for cold storage/occasional batch ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Murali Ramasami. This blog focuses on on-prem to on-prem HDFS replication for HDP clusters using Hortonworks Data Lifecycle Manager (DLM),  an extensible  service built on the Hortonworks DataPlane Platform. Introduction to DLM Data Lifecycle Manager (DLM) delivers on the promise of location-agnostic, secure replication by encapsulating and copying data seamlessly across physical private storage and public cloud environments. This empowers businesses to deliver the right data in the right environment to power the right use cases. DLM v1.1 provides a complete solution to replicate data, metadata and security policies between on-premises and in cloud ... Read More

Feed: Alation. Author: Jason Ma. “The key point is that no organization governs information simply because it can; there has to be a business context, and the increasing realization of this context explains the rise of information stewardship applications.”  – May 2018 Gartner Market Guide for Information Stewardship ApplicationsThe rise of data lakes, IOT analytics, and big data pipelines has introduced a new world of fast, big data. Now, agility and self-service are favored over batch processing and dependency on IT. This new world of analytics has introduced a different set of complexities that have propelled IT organizations to build ... Read More

Feed: Striving for Optimal Performance. The Appendix A of the Using Oracle Autonomous Data Warehouse Cloud guide describes the DBMS_CLOUD package. Unfortunately, it documents only a subset of the subroutines. And, for some of them, the description could also be enhanced. Therefore, while I was testing all the subroutines the DBMS_CLOUD package provides, I took a number of notes. By the end of my tests, I got what I can call my personal reference guide to the package. Since it might help others, here it is… Introduction To interact with other cloud services, the Oracle Autonomous Data Warehouse Cloud service ... Read More

Feed: Microsoft Azure Blog. Author: Tomas Talius. Azure SQL Data Warehouse is a fast, flexible and secure analytics platform for enterprises of all sizes. Today we announced significant query performance improvements for Azure SQL Data Warehouse (SQL DW) customers enabled through enhancements in the distributed query execution layer. Analytics workload performance is determined by two major factors, I/O bandwidth to storage and repartitioning speed, also known as shuffle speed. In this previous blog post, we described how SQL DW caches relevant data to take advantage of NVMe based local storage. In this blog post, we will go under the hood ... Read More

Feed: Planet PostgreSQL. 1st RULE: You do not upgrade PostgreSQL with trigger-based replication2nd RULE: You DO NOT upgrade PostgreSQL with trigger-based replication3rd RULE: If you upgrade PostgreSQL with trigger-based replication, prepare to suffer. And prepare well. There must be a very serious reason to not use the pg_upgrade for upgrading the PostgreSQL. OK, let’s say you can’t afford more than seconds of downtime. Use pglogical then. OK let’s say you run 9.3 and thus can’t use pglogical. Use Londiste. Can’t find readable README? Use SLONY. Too complicated? Use streaming replication - promote the slave and run pg_upgrade on it - ... Read More