Tag: partitions

Feed: Databasejournal.com - Feature Database Articles. Author: . Microsoft HDInsight is the cloud service that deploys and provisions Hadoop clusters on the Azure cloud.  HDInsight is a completely managed and open source analytics service to support enterprise needs. HDInsight supports a wide variety of scenarios with the help of open source frameworks like Hadoop, Storm, Spark, R Server and Hive, not limited to define a pipeline for E-extract, T-transform, L-load, IoT, machine learning, and many more. With enterprise feedback on non-availability of Kafka as a managed service and guaranteeing Kafka bits are up and running, Microsoft included Kafka as service ... Read More

Feed: AWS Big Data Blog. AWS Glue provides enhanced support for working with datasets that are organized into Hive-style partitions. AWS Glue crawlers automatically identify partitions in your Amazon S3 data. The AWS Glue ETL (extract, transform, and load) library natively supports partitions when you work with DynamicFrames. DynamicFrames represent a distributed collection of data without requiring you to specify a schema. You can now push down predicates when creating DynamicFrames to filter out partitions and avoid costly calls to S3. We have also added support for writing DynamicFrames directly into partitioned directories without converting them to Apache Spark DataFrames.Partitioning ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Vinod Sharma. This is our second post in this sub series “Machine Learning Types”. Our master series for this sub series is “Machine Learning Explained”. Unsupervised Learning; is one of three types of machine learning i.e. Supervised Machine Learning, Unsupervised Machine Learning and Reinforcement Learning. This post is limited to Unsupervised Machine Learning to explorer its details. Unsupervised Machine Learning A technique with the idea to explore hidden gems / patterns. To find some intrinsic structure in data. Something cant be seen with naked eye requires magnifier (UML) In Unsupervised Learning available ... Read More

Feed: Planet PostgreSQL. Table of Contents On the last few years, PostgreSQL received several enhancements to process high-volume databases. This first post will try to list them. We will see that they can be splitted in different kinds: Parallelism Improvement of query processing Partitioning Access methods Maintenance tasks SQL features In order to maintain clarity, the explanation of each feature will remain brief. Note: this article was written during the development phase of version 11. I integrated new features of version 11. As long as it is not released, these features might be removed. I thank Guillaume Lelarge for his ... Read More

Feed: Cloud – Cloudera Engineering Blog. Author: Cy Jervis. Self-service BI and exploratory analytics are some of the most common use cases we see our customers running on Cloudera’s analytic database solution. Over the past year, we made significant advancements to provide a more powerful user experience for SQL developers and make them more productive for their everyday self-service BI tasks and workflows. Leveraging Hue as the SQL development workbench, we continue to see usage of the platform increase and the number of analytic use cases grow – with several thousands of weekly users of Hue running multiple million of ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Brandon Wilson. Enterprises are increasingly moving portions or entire datacenters to the cloud in order to minimize their physical footprint, minimize operational overhead, and shorten their infrastructure acquisition cycles. An incidental benefit is that cloud services, like cloud-based object storage, bring a new set of tools to a Hadoop architect. At Hortonworks, our customers use a number of different cloud architectures but we frequently recommend two when it comes to using Apache Hive for analytics and reporting. In this blog, we will present those two architectures, examine their advantages and disadvantages, and provide reproducible ... Read More

Feed: Google Cloud Big Data and Machine Learning Blog. Author: Google Cloud Big Data and Machine Learning Blog Team. By Felipe Hoffa, Developer Advocate What's the simplest way to analyze a 4 terabyte dataset of Wikipedia pageviews made publicly available by the Wikimedia Foundation? (We see this dataset as a public treasure: it’s not perfectly structured, but BigQuery proved an excellent tool to extract the insights we present below.) In this post, I'll show you how to take on this challenge using only BigQuery, while showcasing a few of its newest features. As a lazy programmer, these new features help reduce ... Read More

Feed: MemSQL Blog. Author: Seth Luersen. In a recent MemSQL webcast, we discussed how modern enterprises can easily adopt new data management tools to manage data size, growth, and complexity. Then we demonstrated how to use Apache Kafka and MemSQL to build interactive, real-time data pipelines. Data pipelines capture, process, and serve massive amounts of data to millions of users. During the webcast we also shared how to: Build new data pipelines with modern tools Enable data workflows to support machine learning and predictive analytics with data pipelines Deploy a real-time data pipeline using Apache Kafka and MemSQL Pipelines in ... Read More

Feed: Planet MySQL; Author: Jesper Krogh; Tweet Today – 31 March – is world backup day, so I thought I would write a little about backups in MySQL NDB Cluster. Just because NDB Cluster offers built-in redundancy and high availability does not mean backups are not important. They way – as ever and as for everything in software. The redundancy does not protect against user errors (anyone ever executed DROP TABLE or DROP SCHEMA by accident?) neither does it protect against a natural disaster, fire, or another disaster hitting the data center. Similar with high availability. In short, if the ... Read More

Feed: MemSQL Blog. Author: Jeremy Althouse. In a recent webcast, we shared some tips and tricks on how you can operationalize MemSQL for configuring MemSQL Ops and MemSQL root user passwords; memory settings and health metrics; and how you can take backups and add nodes securely to your cluster. Here are the topics we covered: Permissioning a new cluster by adding a super-user to MemSQL Ops and via GRANT statements on the cluster itself   Best practices for configuring memory limits Best practices for basic cluster monitoring, and how to retrieve the relevant metrics How to take physical backups of ... Read More