Tag: Apache

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. In Part 1 of this blog, we covered some common challenges in memory tuning and baseline setup related to a production Solr deployment. In Part 2, you will learn memory tuning, GC tuning and some best practices. Memory Tuning We assume you have read part 1 of the blog and have a stable Solr deployment up running. The next step is memory tuning to get more out of Solr. Before changing any configuration please be aware that playing with some tuning knobs can cause unexpected consequences on the system, such as ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. Configuring Apache Solr memory properly is critical for production system stability and performance. It can be hard to find the right balance between competing goals. There are also multiple factors, implicit or explicit, that need to be taken into consideration. This blog talks about some common tasks in memory tuning and guides you through the process to help you understand how to configure Solr memory for a production system. For simplicity, this blog applies to Solr in Cloudera CDH5.11 running on top of HDFS. The platform is Oracle JDK 8 and ... Read More

Feed: AWS Big Data Blog. This is a guest post by Dillon Morrison at Looker. Looker is, in their own words, “a new kind of analytics platform–letting everyone in your business make better decisions by getting reliable answers from a tool they can use.” As the breadth of AWS products and services continues to grow, customers are able to more easily move their technology stack and core infrastructure to AWS. One of the attractive benefits of AWS is the cost savings. Rather than paying upfront capital expenses for large on-premises systems, customers can instead pay variables expenses for on-demand services. To further ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Favio Vázquez. Outdated, inaccurate, or duplicated data won’t drive optimal data driven solutions. When data is inaccurate, leads are harder to track and nurture, and insights may be flawed. The data on which you base your big data strategy must be accurate, up-to-date, as complete as possible, and should not contain duplicate entries. Clean data results in better decisions. Cleaning data is the most time-consuming and least enjoyable data science task (until Optimus), but one of the most important ones. No one can start a data science, machine learning or data driven solution without being sure that the ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. Unmodified TPC-DS-based performance benchmark show Impala’s leadership compared to a traditional analytic database (Greenplum), especially for multi-user concurrent workloads. Additionally, benchmark continues to demonstrate significant performance gap between analytic databases and SQL-on-Hadoop engines like Hive LLAP, Spark SQL, and Presto. The past year has been one of the biggest for Apache Impala (incubating). Not only has the team continued to work on ever-growing scale and stability, but a number of key capabilities have been rolled out that further solidifies Impala as the open standard for high-performance BI and SQL analytics. For ... Read More

Feed: All - O'Reilly Media. Author: Jesse Anderson. Oranges and apples (source: Ruth Hartnup on Flickr) Check out Jesse Anderson’s tutorial “Real-time systems with Spark Streaming and Kafka” at the Strata Data Conference, New York City, Sept. 25-28, 2017. Best price ends August 11. If you look up the phrase “boiling the ocean,” it’s defined as writing a post on choosing a cloud provider—there are so many different facets and use cases, and each variable complicates your choice. The key is to narrow the field to your specific situation and needs. In this article, I share some of the early ... Read More

Feed: All - O'Reilly Media. Author: Jon Bruner. Framework (source: Pixabay) After several decades as an interest of academic computer scientists and specialized research labs, deep learning is appearing widely in real products. That transition has led to several exciting new deep-learning frameworks that tend to emphasize either rapid prototyping or efficient deployment at scale. For product developers looking to experiment with an idea and then reinforce and deploy it, a single framework that supports both ends of the process is helpful. Apache MXNet, an open-source deep learning framework first published in 2015, aims to achieve exactly that. I recently ... Read More

Feed: All - O'Reilly Media. Author: Manu Jeevan, Suresh Chelladurai. Road signs (source: Pixabay) The full Jupyter Notebook for this post can be accessed here. Although there are many deep learning frameworks, including TensorFlow, Keras, Torch, and Caffe, Apache MXNet in particular is gaining popularity due to its scalability across multiple GPUs. In this blog post, we'll tackle a computer vision problem: classifying German traffic signs using a convolutional neural network. The network takes a color photo containing a traffic sign image as input, and tries to identify the type of sign. The full notebook can be found here In ... Read More

Feed: Neo4j Graph Database. Author: Rachel Howard. By We analyzed the data and created a Gephi map of the 15,000 top users. One of the clusters we identified included journalists, the most prominent of whom was Washington Post reporter Because there was no budget, I had to use open source software and started with The original source code — the Python scripts and important routines for pulling this into Neo4j — is also still available on The key thing we learned in the research is that people are not disciplined about following the process of collecting data. Instead we typically ... Read More

Feed: CDH – Cloudera Engineering Blog. Author: Cy Jervis. Analytical and operational access patterns are very different and until now the Hadoop ecosystem has not had a single storage engine that could support both. As a result, engineers have been forced to implement complex architectures that stitch multiple systems together in order to provide these capabilities. On one hand immutable data on HDFS offers superior analytic performance, while mutable data in Apache HBase is best for operational workloads. Apache Kudu bridges this gap.   Kudu’s architecture is shaped towards the ability to provide very good analytical performance, while at the ... Read More