Tag: partition

Feed: Microsoft Azure Blog. Author: Rob Caron. Highlights New Azure innovation advances customer success for the cloud- and AI-powered future - In preparation for Microsoft Inspire in Las Vegas this week (where partners and Microsoft meet to connect, learn, and collaborate to accelerate the digital transformation and success of our customers), Jason Zander (Executive Vice President, Microsoft Azure) highlighted new offers and technologies that will help organizations navigate their own digital transformation and build the critical foundations for cloud and AI. He showcases momentum from businesses and partners who are investing in intelligent tech from the cloud to the edge ... Read More

Feed: Planet MySQL; Author: Severalnines; One of the cool features in Galera is automatic node provisioning and membership control. If a node fails or loses communication, it will be automatically evicted from the cluster and remain unoperational. As long as the majority of nodes are still communicating (Galera calls this PC - primary component), there is a very high chance the failed node would be able to automatically rejoin, resync and resume the replication once the connectivity is back. Generally, all Galera nodes are equal. They hold the same data set and same role as masters, capable of handling read ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Saumitra Buragohain. We’re excited to announce the long-awaited release of Hortonworks Data Platform 3.0.0. HDP 3.0 is faster, smarter, hybrid, bigger, trusted and real-time database. We encourage you to read our blog that announced HDP 3.0. You can also view our keynote presentation from DataWorks Summit, where we demoed an autonomous car (1/10 scale), trained with HDP 3.0 technologies. We also want to thank  our current and prospective customers and partners for signing up for the HDP 3.0 Early Access and we are encouraged by the huge interest- now, you can use the HDP ... Read More

Feed: Microsoft Azure Blog. Author: Dhruv Goel. Data engineers love Kafka on HDInsight as a high-throughput, low-latency ingestion platform in their real time data pipeline. They already leverage Kafka features such as message compression, configurable retention policy, and log compaction. With the release of Apache Kafka 1.0 on HDInsight, customers now get key features that make it easy to implement the most demanding scenarios. Here is a quick introduction: Idempotent producers so that you don’t have to deduplicate Consider a cellular billing system, in which the producer writes the amount of data consumed by users to a Kafka topic called ... Read More

Feed: Neo4j Graph Database Platform. Author: Jocelyn Hoppa. Editor’s Note: This presentation was given by Johannes Unterstein at GraphConnect Europe in May 2017.Presentation SummaryIn this talk, Johannes Unterstein walks you through the mechanics and advantages of using Apache Mesos and DC/OS (data center operating system) for hosting your Neo4j Causal Cluster.Unterstein starts by addressing the question of whether or not you even need a cluster (yes, you probably do). He then moves on to discuss how to keep data persisted through node failures, and how the best way to protect your data is through clusters of servers, including the Causal ... Read More

Feed: Planet PostgreSQL. This blog is about my work on the Postgres open source database, and is published on Planet PostgreSQL. PgLife allows monitoring of all Postgres community activity. Oracle Real Application Clusters (RAC) Monday, July 9, 2018 I get asked about Oracle RAC often. My usual answer is that Oracle RAC gives you 50% of high reliability (storage is shared, mirroring helps) and 50% of scaling (CPU and memory is scaled, storage is not). The requirement to partition applications to specific nodes to avoid cache consistency overhead is another downside. (My scaling presentation shows Oracle RAC.) I said the ... Read More

Feed: R-bloggers. Author: John Mount. In this note we will show how to speed up work in R by partitioning data and process-level parallelization. We will show the technique with three different R packages: rqdatatable, data.table, and dplyr. The methods shown will also work with base-R and other packages. For each of the above packages we speed up work by using wrapr::execute_parallel which in turn uses wrapr::partition_tables to partition un-related data.frame rows and then distributes them to different processors to be executed. rqdatatable::ex_data_table_parallel conveniently bundles all of these steps together when working with rquery pipelines. The partitioning is specified by ... Read More

Feed: Cloudera Manager – Cloudera Engineering Blog. Author: Justin Kestelyn. StackIQ takes a “software defined infrastructure” approach to provision and manage cluster infrastructure that sits below Big Data platforms such as Apache Hadoop. In the guest post below, StackIQ co-founder and VP Engineering Greg Bruno explains how to install Cloudera Enterprise on top of StackIQ’s management system so they can work together. The hardware used for this deployment is a small cluster: one node (i.e. one server) for the StackIQ Cluster Manager and four nodes as backend/data nodes. Each node has two disks and all nodes are connected via 1Gb ... Read More

Feed: Planet PostgreSQL. Recently, someone reached out to me asking “what’s the best way to achieve database DML auditing in Postgres?” I have suggested a couple of options below to achieve DML auditing in Postgres: Audit using Postgres logs. Postgres allows to keep track of DML statements at database level by enabling auditing directly in the postgres logs. You can use https://github.com/pgaudit/pgaudit extension to make the audit logging consistent and readable. Audit tables using trigger based approach by creating audit schema on live database and keep the audit tables updated through trigger. Both of these approaches have pros and cons. If you are looking for detailed auditing ... Read More

Feed: AWS Big Data Blog. At Goodreads, we’re currently in the process of decomposing our monolithic Rails application into microservices. For the vast majority of those services, we’ve decided to use Amazon DynamoDB as the primary data store. We like DynamoDB because it gives us consistent, single-digit-millisecond performance across a variety of our storage and throughput needs.Although DynamoDB excels at high-throughput read and write workloads, it’s not optimized to support one-time, ad hoc queries or data warehouse workloads. However, by combining AWS Data Pipeline, Amazon S3, AWS Glue, and Amazon Athena we can export our dataset from DynamoDB to S3 ... Read More