Category: Hortonworks

Feed: Hortonworks Blog – Hortonworks. Author: George Vetticaden. IN HDF 3.1, we announced support for Apache Kafka 1.0 with powerful integrations with Apache Ambari, Apache Ranger, and Apache Atlas. In the last 12 months, we have seen Kafka emerge as a key component in many of our customers streaming architectures’. A common architecture looks something like the following: Different customers or even different groups within the same organization have different approaches to do analytics on the stream but Kafka is constant, it’s everywhere, and is a key component in many of our customers’ architectures. Kafka Blindness Over the last 12 months, the ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Amol Thacker. This is the finale of the blog series (see part 1, part 2, part 3) where having discussed the problem domain, looked at the functional and architectural aspects and prepared the environment, we are now ready to execute a few pricing calculations. SSH into the cluster gateway node and download the following from repo: compute/compute-engine-spark-1.0.0.jar compute/scripts/compute-price.sh Notice the directives around using Docker as executor env for Spark on YARN in client mode. You should now be ready to simulate a distributed pricing compute using the following command: ./compute-price.sh where metric: FwdRate: Spot Price ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Amol Thacker. This is the 3nd blog in the series (see part 1, part 2) where we will walk through the tech stack and prepare the environment. The entire infrastructure is provisioned on OpenStack private cloud using Cloudbreak 2.7.0 which first automates the installation of Docker CE 18.03 on CentOS 7.4 VMs and then the installation of HDP 3.0 cluster with Apache Hadoop 3.1 and Apache Spark 2.3 using the new shiny Apache Ambari 2.7.0 To Try This at Home Ensure you have access to a Hortonworks Cloudbreak 2.7.0 instance. Please refer to the ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Matt Spillar. Earlier this week, Hortonworks announced financial results for the second quarter of 2018. Momentum is high at the halfway point of the year, as our CEO Rob Bearden announced another record breaking quarter! The Q2 revenue of $86.3 million marked a total revenue growth of 40 percent compared to the prior year. Q2 also was another quarter of positive operating cash flow. Hortonworks has been a public company for over four years now, and in that time, has evolved from the leading Hadoop provider to the leading global data management platform company. More enterprises ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Amol Thacker. This is the 2nd blog in a 4-part blog series (see part 1) where we will dive into representative pricing semantics and architectural aspects of a prototype implementation using HDP 3.0.  Pricing Semantics  The engine leverages QuantLib, an open source library for quantitative finance, to compute: Spot Price of a Forward Rate Agreement (FRA) using the library’s yield term structure based on flat interpolation of forward rates. NPV of a vanilla fixed-float 5yr Interest Rate Swap (IRS) NPV of a European Equity Put Option averaged over multiple algorithmic calculations (Black-Scholes, Binomial, Monte-Carlo) For ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Rohith Sharma KS. Introduction Lifetime indicates the overall time spent by an application in YARN. The lifetime of an application is calculated from its start time to finish time, including the actual run time as well as the wait time for resource allocation. Both users and administrators on the YARN system might occasionally be required to restrict the duration of specific lifetime Service Level Agreements (SLAs). The requirements for restricting the SLA durations are different for users and administrators. For example, a user might run a scheduled cron job that returns statistics of an ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Ram Venkatesh. This post was authored by Ram Venkatesh, Hortonworks VP, Engineering; and James Malone, Cloud Product Manager, Google.  If you’re looking for a fully-managed cloud service for running data and analytics clusters, thanks in large part to the Apache Hadoop and Spark communities, you might very well look to Cloud Dataproc, which offers both long-running and job-scoped clusters. Job-scoped clusters let you tailor your clusters to a specific job, instead of spending development time building individual job configurations that compete for resources on a multi-purpose cluster. Dataproc even has features such as workflow ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Amol Thacker. This is the first blog in a 4-part blog series where we will look at an architectural approach to implementing a distributed compute engine for pricing financial derivatives using Hortonworks Data Platform [HDP] 3.0. In this blog, we will discuss the problem domain and set the context before we zoom in on the functional and technical aspects. Modern financial trading and risk platforms employ compute engines for pricing and risk analytics across different asset classes to drive real-time trading decisions and quantitative risk management. Pricing financial instruments involves a range of algorithms ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Brent Biddulph. The traditional retail business model is evolving rapidly right before our eyes, as early retail e-commerce sites once effectively emulated stores with the assumption that consumers would begin and end their shopping trip in that channel, just as they did in the store.  Fast forward to today, and all that has changed in the digital economy we all are now part of – where supply chain ‘networks’ are evolving into more complex fulfillment ‘nodes’ in an omni-channel retail business model striving for profitable outcomes. As Brian Kilcourse, Managing Partner at RSR in ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Sunil Govindan. Meetup Link: https://www.meetup.com/Bangalore-Hadoop-Meetups/events/252534327/ The Bangalore Apache Hadoop Meetup group, with over 3400 members who share common interests and ideas in the Hadoop ecosystem, brings together a community of practitioners and developers at Bangalore. The talks at this meetup cover a variety of topics related to the Hadoop ecosystem, such as Data Science workloads, Big Data-Driven Applications, SQL on Hadoop, YARN workloads — the list is endless. The most recent meetup of the group was held on July 28, 2018 at LinkedIn, Bangalore. Over a hundred enthusiastic participants attended this meetup on a ... Read More