Tag: Cache

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. In Part 1 of this blog, we covered some common challenges in memory tuning and baseline setup related to a production Solr deployment. In Part 2, you will learn memory tuning, GC tuning and some best practices. Memory Tuning We assume you have read part 1 of the blog and have a stable Solr deployment up running. The next step is memory tuning to get more out of Solr. Before changing any configuration please be aware that playing with some tuning knobs can cause unexpected consequences on the system, such as ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. Configuring Apache Solr memory properly is critical for production system stability and performance. It can be hard to find the right balance between competing goals. There are also multiple factors, implicit or explicit, that need to be taken into consideration. This blog talks about some common tasks in memory tuning and guides you through the process to help you understand how to configure Solr memory for a production system. For simplicity, this blog applies to Solr in Cloudera CDH5.11 running on top of HDFS. The platform is Oracle JDK 8 and ... Read More

Feed: R-bloggers. Author: hrbrmstr. I’ve blathered about my crawl_delay project before and am just waiting for a rainy weekend to be able to crank out a follow-up post on it. Working on that project involved sifting through thousands of Web Archive (WARC) files. While I have a nascent package on github to work with WARC files it’s a tad fragile and improving it would mean reinventing many wheels (i.e. there are longstanding solid implementations of WARC libraries in many other languages that could be tapped vs writing a C++-backed implementation). One of those implementations is JWAT, a library written in ... Read More

Feed: AWS Government, Education, & Nonprofits Blog. Author: publicsector. The Intelligence Advanced Research Projects Activity (IARPA) Machine Intelligence from Cortical Networks (MICrONS) program seeks to revolutionize machine learning by better understanding the representations, transformations, and learning rules employed by the brain.We spoke with Dean Kleissas, Research Engineer working on the IARPA MICrONS Project at the Johns Hopkins University Applied Physics Laboratory (JHU/APL), and he shared more about the project, what makes it unique, and how the team leverages serverless technology. Could you tell us about the IARPA MICrONS Project? This project partners computer scientists, neuroscientists, biologists, and other researchers from ... Read More

Feed: All - O'Reilly Media. Author: Andrew Odewahn. JupyterHub + Kubernetes (source: O'Reilly) Meet the team behind JupyterHub at JupyterCon on August 22-25, 2017, in New York City. Registration is now open. JupyterHub, a "multi-user server for Jupyter Notebooks," is an essential tool for teaching and training at scale with Jupyter. As described in The course of the future – and the technology behind it , JupyterHub is being used to power an introductory class in data science taken by hundreds of students at Berkeley every semester. JupyterHub is a complex piece of software, and setting up and operating has been ... Read More

Feed: Google Cloud Big Data and Machine Learning Blog. Author: Google Cloud Big Data and Machine Learning Blog Team. Innovation in data processing and machine learning technology Monday, August 21, 2017 By Reza Rokni and John LaBarge, Solution Architects This open-ended series (see first installment) documents the most common patterns we’ve seen across production Cloud Dataflow deployments. In Part 2, we’re bringing you another batch — including solutions and pseudocode for implementation in your own environment. Let’s dive in! Pattern: GroupBy using multiple data properties Description: Data elements need to be grouped by multiple properties. Example: IoT data arrives with ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. Analytical and operational access patterns are very different and until now the Hadoop ecosystem has not had a single storage engine that could support both. As a result, engineers have been forced to implement complex architectures that stitch multiple systems together in order to provide these capabilities. On one hand immutable data on HDFS offers superior analytic performance, while mutable data in Apache HBase is best for operational workloads. Apache Kudu bridges this gap.   Kudu’s architecture is shaped towards the ability to provide very good analytical performance, while at the ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Justin Kestelyn. Released with CDH 5.8, Impala 2.6 brings solid performance improvements, particularly for clusters secured by Kerberos running BI workloads on Apache Hadoop. Just a few months back, we showed you how Impala 2.5 delivered a 4x performance boost compared to Impala 2.3 for BI workloads on Hadoop via the introduction of several features like runtime filters. Here’s an update: Compared to two releases ago, Impala 2.6 delivers 12x better performance on secure workloads and continues this drumbeat of consistent performance improvement. We are excited to share details on performance improvements in ... Read More

Feed: Planet MySQL; Author: Keith Hollman; I’ve had some questions from people using MySQL Cluster GPL and wanting to move to using MySQL Cluster Carrier Grade Edition, i.e., they want to use MySQL Cluster Manager, MCM, to make their lives much easier, in particular, upgrading (as well as config change ease and backup history). All I want to do here is to share with you my personal experience on migrating what’s considered a ‘wild’ NDB Cluster to a MCM managed cluster. It’s just as simple to follow the manual chapter Importing a Cluster into MySQL Cluster Manager so at least ... Read More

Feed: R-bloggers. Author: David Smith. Making your code run faster is often the primary goal when using parallel programming techniques in R, but sometimes the effort of converting your code to use a parallel framework leads only to disappointment, at least initially. Norman Matloff, author of Parallel Computing for Data Science: With Examples in R, C++ and CUDA, has shared chapter 2 of that book online, and it describes some of the issues that can lead to poor performance. They include: Communications overhead, particularly an issue with fine-grained parallelism consisting of a very large number of relatively small tasks; Load balance, where ... Read More