Tag: cluster

Feed: Cloudera Engineering Blog » Hadoop. Author: Aaron Kimball. Configuring a Hadoop cluster is something akin to voodoo. There are a large number of variables in hadoop-default.xml that you can override in hadoop-site.xml. Some specify file paths on your system, but others adjust levers and knobs deep inside Hadoop’s guts. Unfortuately, there’s little or no documentation on how to set them well. Is there a single optimal configuration? Are there some settings that can just be “set to 11?” At Cloudera, we’re working hard to make Hadoop easier to use and to make configuration less painful. Our Hadoop Configuration Tool ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Philip Zeyliger. Hadoop’s NameNode, SecondaryNameNode, DataNode, JobTracker, and TaskTracker daemons all expose runtime metrics. These are handy for monitoring and ad-hoc exploration of the system and provide a goldmine of historical data when debugging. In this post, I’ll first talk about saving metrics to a file.  Then we’ll walk through some of the metrics data.  Finally, I’ll show you how to configure sending metrics to other systems and explore them with jconsole. Dumping metrics to a file The simplest way to configure Hadoop metrics is to funnel them into a user-configurable file on ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Aaron Kimball. You might think that the SecondaryNameNode is a hot backup daemon for the NameNode. You’d be wrong. The SecondaryNameNode is a poorly understood component of the HDFS architecture, but one which provides the important function of lowering NameNode restart time. This blog post describes how to configure this daemon in a large-scale environment. The default Hadoop configuration places an instance of the SecondaryNameNode on the same node as the NameNode. A more scalable configuration involves configuring the SecondaryNameNode on a different machine. About the SecondaryNameNode The NameNode is responsible for the ... Read More

Feed: Cloudera Engineering Blog » Hadoop. Author: Tom White. It’s a new year, the time when we take a moment to look back at the previous one, and forward to what might be coming next. In the world of Hadoop a lot happened in 2008. Organization At the beginning of the year, Hadoop was a sub-project of Lucene. In January, Hadoop became a Top Level Project at Apache, in recognition of its success and diversity of community. This allowed sub-projects to be added, the first of which was HBase, previously a contrib project. ZooKeeper, a service for coordinating distributed systems, ... Read More

Feed: Big Data Made Simple - One source. Many perspectives.. Author: Baiju NT. In the world of Big Data, information has evolved from being a merely passive construct into an active and dynamic abstraction. The language of this information is the computer code, but before diving into cluster programming, there are several things that you must first understand. Amdahl’s Law and the Limits of Parallel Computation Despite the power and promise of parallel computation, there are limits to what parallel computers can accomplish. In 1967, computer scientist Gene Amdahl presented a paper at that year’s AFIPS Spring Joint Computer Conference ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. An overview of YARN integration with Big SQL was presented in a previous blog. Note that this is NOT a performance feature, if you are solely interested in improving performance then logical Big SQL workers should be chosen instead of enabling YARN for Big SQL. When YARN is enabled for Big SQL, there is more flexibility to increase the resources for Big SQL without needing to restart the Big SQL service. Read more about Big SQL Yarn Configuration options and recommendations and how to enable YARN for Big SQL. Scheduling control is taken away ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. When YARN is enabled for Big SQL, YARN controls how much memory and CPU resources are assigned to Big SQL at any given point in time. Other applications such as Spark or Hive can continue to use YARN resources when YARN is enabled for Big SQL. Read more about Big SQL 4.2.5 and YARN integration. There are several YARN and Big SQL configuration parameters that are at play with YARN is enabled with Big SQL. These configuration parameters and best practice to setting these parameter are discussed in this blog. YARN Container Memory Settings ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Roni Fontaine. On June 7, we hosted the Next Gen Data Analytics Powered by Cloud webinar with speakers from Hortonworks, Jeff Sposetti  and Amazon Web Services, Karthik Krishnan. The webinar provided an overview on how your organization can achieve the benefits of data analytics with the cloud, how to use the AWS marketplace for ease of deployment, and how to leverage Hortonworks cloud solutions for data science and analytics needs. Jeff and Karthik provided demos to show how easy it is to get started in the cloud. To get access to the slides, go ... Read More

Feed: Microsoft Azure Blog. Author: Asir Selvasingh. We released 1.1 of the Azure Management Libraries for Java. This release adds support for: Cosmos DB Azure Container Service and Registry Active Directory Graphhttps://github.com/Azure/azure-sdk-for-java Getting started Add the following dependency fragment to your Maven POM file to use 1.1 version of the libraries: com.microsoft.azure azure 1.1.0 Create a Cosmos DB with DocumentDB API You can create a Cosmos DB account by using a define() … create() method chain.DocumentDBAccount documentDBAccount = azure.documentDBs().define("myDocumentDB") .withRegion(Region.US_EAST) .withNewResourceGroup(rgName) .withKind(DatabaseAccountKind.GLOBAL_DOCUMENT_DB) .withSessionConsistency() .withWriteReplication(Region.US_WEST) .withReadReplication(Region.US_CENTRAL) .create(); In addition, you can: Create Cosmos DB with DocumentDB API and configure for high ... Read More

Feed: Neo4j Graph Database. Author: Rachel Howard. By Dippy Aggarwal, PhD Candidate, University of Cincinnati | June 22, 2017  Editor’s Note: This presentation was given by Dippy Aggarwal at GraphConnect San Francisco in October 2016.Presentation SummaryContainer orchestration for multiple containers across a fleet of machines has the potential to solve issues across the range of scaling, replication, fault tolerance and container communication.In this GraphConnect presentation, Dippy Aggarwal walks you through the basics of Docker containerization and why Docker alone isn’t enough to handle today’s challenges of scale and multi-cluster deployments. The three most popular container orchestration tools she examines include ... Read More