Tag: partitions

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. This framework based on Apache Flume, Apache Spark Streaming, and Apache Impala (incubating) can detect and report on abnormal bad HTTP requests within seconds.                      Website performance and availability are mission-critical for companies of all types and sizes, not just those with a revenue stream directly tied to the web. Web pages can become unavailable for many reasons, including overburdened backing data stores or content-management systems or a delay in load times of third-party content such as advertisements. In addition to the deterioration of user experience during such incidents, search engines quickly ... Read More

Feed: R-bloggers. Author: msuzen. Preamble There is a lot of confusion among practitioners regarding the concept of overfitting. It seems like, a kind of an urban legend or a meme, a folklore is circulating in data science or allied fields with the following statement:Applying cross-validation prevents overfitting and a good out-of-sample performance, low generalisation error in unseen data, indicates not an overfit. This statement is of course not true: cross-validation does not prevent your model to overfit and good out-of-sample performance does not guarantee not-overfitted model. What actually people refer to in one aspect of this statement is called overtraining. Unfortunately, this meme is not only ... Read More

Feed: Planet big data. Author: Meg Blanchette. August 15th, 2017 An important architectural component of any data platform is those pieces that manage data ingestion. In many of today’s “big data” environments, the data involved is at such scale in terms of throughput (think of the Twitter “firehose”) or volume (e.g., the 1000 Genomes project) that approaches and tools must be carefully considered. In the last few years, Apache Kafka and Apache Spark have become popular tools in a data architect’s tool chest, as they are equipped to handle a wide variety of data ingestion scenarios and have been used ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Carter Shanklin. This is part 1 of a 2 part series for how to update Hive Tables the easy way Historically, keeping data up-to-date in Apache Hive required custom application development that is complex, non-performant and difficult to maintain. HDP 2.6 radically simplifies data maintenance with the introduction of SQL MERGE in Hive, complementing existing INSERT, UPDATE and DELETE capabilities. This blog shows how to solve common data management problems, including: Hive upserts, to synchronize Hive data with a source RDBMS. Update the partition where data lives in Hive. Selectively mask or purge data ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. Impala 2.5, now shipping in CDH 5.7, brings significant performance improvements and some highly requested features. Impala has proven to be a high-performance analytics query engine since the beginning. Even as an initial production release in 2013, it demonstrated performance 2x faster than a traditional DBMS, and each subsequent release has continued to demonstrate the wide performance gap between Impala’s analytic-database architecture and SQL-on-Apache Hadoop alternatives. Today, we are excited to continue that track record via some important performance gains for Impala 2.5 (with more to come on the roadmap), summarized below ... Read More

Feed: SAS Blogs. Author: Stephen Foerster. The Cloud Analytic Server (CAS for short) is SAS’ latest high-performance, scalable, in-memory analytic data server.  In this post, I’d like to discuss the CAS physical data model, i.e.what features CAS offers for data storage, and how to use them to maximize performance in CAS (and consequently SAS Visual Analytics 8.1 too). So, specifically let’s answer the question: What CAS physical table storage features can we use to get better performance in CAS and SAS Visual Analytics/CAS? CAS Physical Table Storage Features The following data storage features affect how CAS tables are physically structured: ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. In this installment of our series about how quality assurance is done at Cloudera, learn about the important role of fault injection in the overall process. Apache Hadoop is the consummate example of a scalable distributed system (SDS); such systems are designed to provide 24/7 services reliably and to scale elastically with the addition of industry-standard hardware cost-effectively. They must be resilient and fault-tolerant to various environmental anomalies. As you would expect, the software layer in Hadoop that provides risk-aware management is quite complex. Thus, for customers running Hadoop in production, severe ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. In this post, engineers from Wargaming.net, the online game developer and publisher, describe the design of their real-time recommendation engine built on CDH. The scope of activities at Wargaming.net extends far beyond the development of games. We work on dozens of internal projects simultaneously, and our Data-driven Real-time Rules Engine (DDRRE) is among the most ambitious. DDRRE is a system that analyzes large amounts of data in real time, allowing us to personalize game interaction with players through recommendations, without unwanted advertising or promotional messages. Those recommendations are based on the ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. Cloudera Enterprise 5.5 improves the life of the admin through a deeper integration between HUE and Cloudera Manager, as well as a rebase on HUE 3.9. Cloudera Enterprise 5.5 contains a number of improvements related to HUE (the open source GUI that makes Apache Hadoop easier to use), including easier setup for HUE HA, built-in activity monitoring for improved stability, and better security and reporting via Cloudera Navigator and Apache Sentry (incubating). In this post, we’ll offer an overview of some of these improvements. Automatic HA and Load Balancing With this ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Nadeem Asghar. Attunity is a long-time Hortonworks partner who provides data optimization and data integration software to help Hortonworks customers address exploding data growth, efficiently manage the performance of BI and data warehouse systems, and realize the tremendous economies of Apache Hadoop®. Attunity solutions are certified on HDF, HDP and are YARN Ready. Together, Hortonworks and Attunity are committed to advancing Hadoop through community-led innovation. This new solution is one more example of that. Attunity Compose for Hive By Jordan Martz, Director of Technology Solutions, Attunity Attunity Compose for Hive automates the data pipeline ... Read More