Category: IBM

Feed: Hadoop Dev. Author: Nailah Bissoon. There are several ingestion techniques that can be used to add data to Big SQL and Hive. This blog will give an overview of the various flavors of adding files or appending to files already in HDFS and the commands to execute from either Hive or Big SQL. The technique of adding files directly to HDFS can offer one of the fastest ways to ingest data into Hadoop. Create External Hadoop Tables If files already reside on HDFS, Big SQL tables can be created with the location of these files specified. These tables are ... Read More

Feed: Data Henrik; Author: Henrik Loeser (data_henrik); External Tables - New in Db2 Db2 Warehouse recently added a new feature named external table. Because I was interested in that feature for a while and I have an instance of Db2 Warehouse on Cloud running in the IBM Cloud datacenter in Frankfurt, Germany, I performed some quick tests. Here is what it is and how I got it to work. Overview: External Tables According to the Db2 documentation ... Read More

Feed: Hadoop Dev. Author: JamieNisbet. Introduction In the first part of this blog (Big SQL Problem Determination Part 1), we discussed how to classify a problem into some high-level categories (Crash, Error, Performance, Sys Admin, and I Don’t Know), and then introduced the bigsql-support.sh data collection tool to automate the collection of Big SQL diagnostic data from a cluster depending on the type of problem. In part 2 of this blog, we’ll focus on a performance problem or if an application does not appear to be responsive (hang). These types of performance/hang problems are a different beast and require a ... Read More

Feed: Hadoop Dev. Author: JamieNisbet. Introduction Problem determination is not simple. The solution to a problem might be simple (if you’re lucky!), but going from PROBLEM ==> SOLUTION is where the time and effort is spent! Let’s put on our detective hats! In order for a detective to solve a crime, they need evidence and clues. What information do you need in order to be able to properly investigate and solve a problem in Big SQL? As a frequent investigator of diagnostics myself, often I do not know what information I need until I need it! We cannot always predict ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. An overview of YARN integration with Big SQL was presented in a previous blog. Note that this is NOT a performance feature, if you are solely interested in improving performance then logical Big SQL workers should be chosen instead of enabling YARN for Big SQL. When YARN is enabled for Big SQL, there is more flexibility to increase the resources for Big SQL without needing to restart the Big SQL service. Read more about Big SQL Yarn Configuration options and recommendations and how to enable YARN for Big SQL. Scheduling control is taken away ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. When YARN is enabled for Big SQL, YARN controls how much memory and CPU resources are assigned to Big SQL at any given point in time. Other applications such as Spark or Hive can continue to use YARN resources when YARN is enabled for Big SQL. Read more about Big SQL and YARN integration. There are several YARN and Big SQL configuration parameters that are at play with YARN is enabled with Big SQL. These configuration parameters and best practice to setting these parameter are discussed in this blog. YARN Container Memory Settings The ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. YARN is a central resource manager component which manages and allocates resources to applications running on Hadoop. Multiple applications can run on Hadoop and they can share a common resource manager via YARN. Apache Slider allows for non-YARN services to be managed by YARN. Big SQL is integrated with YARN through Apache Slider since Big SQL 4.2.5 (improvements have been made in Big SQL 5.0.1 in this area). When a component requests resources from YARN, it is put on a YARN priority queue and YARN schedules the allocation based on priority. When the request ... Read More

Feed: Hadoop Dev. Author: Nailah Bissoon. There were so many improvements to Analyze in Big SQL 4.2 that we decided to create a new version number for it. Analyze v2 is now the default in v4.2 read more about Issues resolved with Analyze v2. Analyze v1 refers to the version that depends on the MapReduce framework and Analyze v2 brings all processing into Big SQL. The areas of improvement made to analyze and the performance improvements we measured internally with the new Analyze will be revealed in this blog. The highlights of Analyze improvements since Big SQL 4.2 are: Removes ... Read More

Feed: Hadoop Dev. Author: Shay Roe. IntroductionThis blog is part of a series outlining all you need to know to start working with Big SQL’s Automatic Catalog Synchronization (Auto-Sync). Part 1 of the series provided an introduction to Auto-Sync, discussing its significance, the problem it addresses and how it can be enabled/disabled in Ambari. Part 2 presented a high-level view of the Auto-Sync architecture and provided details on the feature’s main configuration parameters. In this third and final blog of the series, we’ll take a look at problem determination and explain what you can do if experiencing issues related to ... Read More

Feed: Hadoop Dev. Author: JessicaLeeYau. Announcing the immediate availability of Big SQL v5.0.1 – maintenance release Big SQL, a SQL engine on Hadoop, has been making strides with the fast-evolving open source ecosystem. The core capabilities of Big SQL focusses on federation, SQL compatibility, scalability, performance, and of course enterprise security, making it a desirable query engine to seek insights from disparate data sources including Hadoop. We announced Big SQL v5.0 in July, 2017 following our partnership announcement with Hortonworks. Following that, we now announce the release of Big SQL v5.0.1 which focuses on  consumability  (support for – Zeppelin notebook, ... Read More