Tag: Spark

Feed: James Serra's Blog. Author: James Serra. Below are the top 15 questions I am seeing from customers looking to build a modern data warehouse in the cloud, and the blogs that I have wrote that try to answer each question (I have updated most of these blogs): Do I need a data lake? Data lake detailsDo I need a relational data warehouse? Is the traditional data warehouse dead?Do I need a cube? The need for having both a DW and cubesWhere do I clean my data? Where should I clean my data?What is the Common Data Model (CDM)? Common ... Read More

Feed: Microsoft Azure Blog. Author: Jamal Mohammed. This post was co-authored by Shubha Vijayasarathy, Program Manager, Azure Messaging (Event Hubs) With billions of miles logged, MileIQ provides stress-free logging and accurate mileage reports for millions of drivers. Logging and reporting miles driven is a necessity for independent contractors to organizations with employees who need to drive for work. MileIQ automates mileage logging to create accurate records of miles driven, minimizing the effort and time needed with manual calculations. Real-time mileage tracking produces over a million location signal events per hour, requiring fast and resilient event processing that scales. MileIQ leverages ... Read More

Feed: Microsoft Power BI Blog | Microsoft Power BI. Author: . Today, we are announcing general availability (GA) of one of the most significant scalability features in Microsoft BI, Aggregations. Aggregations empower organizations to enable every user to perform their own analysis atop full fidelity, petabyte scale datasets. At the same time, those organizations now have better control over their BI architecture, striking a better balance between cost, performance, and data fidelity. In addition, we are announcing that aggregations can be used for enterprise-grade semantic models in large organizations with fine-grain security requirements by leveraging Power BI’s row level security ... Read More

Feed: IBM Big Data & Analytics Hub - All Content; Author: holly-vatter; Capitalogix is a hedge fund, but it’s really a data science firm in disguise. They work to understand and exploit capital markets by building custom data science models that can analyze massive amounts of data from as many sources as possible. Capitalogix’s need for high-performance analytics and data science on a single platform led them to transition from Microsoft SQL Server to Netezza and finally to the new IBM Integrated Analytics System (IAS). The IAS combines data science, analytic workloads and hybrid cloud capabilities into a single, game-changing ... Read More

Feed: Hadoop – Cloudera Engineering Blog. Author: Wangda Tan. Hello world, it’s been a while! We are super excited today to announce the open-sourcing of one of the exciting new projects we’ve been working behind the scenes at the intersection of big-data and computation platforms – YuniKorn! Yunikorn is a new standalone universal resource-scheduler responsible for allocating/managing resources for big-data workloads including batch jobs and long-running services. Let’s dive right in! Introduction YuniKorn is a light-weight, universal resource scheduler for container orchestrator systems. It is created to achieve fine-grained resource sharing for various workloads efficiently on large scale, multi-tenant environments ... Read More

Feed: Cloud – Cloudera Engineering Blog. Author: Shelby Khan. Fig 1 – Architecture Introduction Many of Cloudera’s customers set up Cloudera Manager to collect their clusters’ diagnostic data on a regular schedule and automatically send that data to Cloudera. Cloudera analyzes this data, identifies potential problems in the customer’s environment, and alerts customers, requiring fewer back-and-forths with our customers when they file a support case and provides Cloudera with critical information to improve future versions of all of Cloudera’s software. If Cloudera discovers a serious issue, Cloudera searches this diagnostic data and proactively notifies Cloudera customers who might encounter problems ... Read More

Feed: R-bloggers. Author: Hadley Wickham. rstudio::conf, the conference for all things R and RStudio, will take place January 29 and 30, 2020 in San Francisco, California. It will be preceded by Training Days on January 27 and 28. Early Bird registration is now open! Conference: Wednesday-Thursday, Jan 29-30 Join me, your host and Chief Scientist of RStudio, for our keynote speakers: Along with 80 other talks in four parallel tracks. As well as RStudio data scientists and engineers, you’ll also hear from and interact with outstanding speakers drawn from the wider R and data science communities. Stay tuned for announcements ... Read More

Feed: Planet big data. Author: Andrew Hogan. Is your CMO considering shifting investments in marketing technology toward the services of creative agencies as Forrester recommends? You should find out — because one of two possible scenarios will play out.   The winning scenario: If you 1) boost your internal design team and better integrate brand, 2) contract with firms that apply human-centric design, and 3) apply design thinking effectively — the real thing, not the shallow variety — then your company will differentiate effectively and win.   The losing scenario: Without these three ingredients, increasing your agency spending will deliver the wrong kind of differentiation: experiences ... Read More

Feed: MemSQL Blog. Author: Jacky Liang. MemSQL Forums are available anytime you have a question about MemSQL, want to know more about MemSQL, or are willing and able to help others. And now the MemSQL Forums, coming up on their one-year anniversary, have their first Community Star: Ziv Meidav, a big data architect. The community of MemSQL users, developers, and employees gather at the MemSQL Forums to ask questions, offer support, and discuss strategies for leveraging MemSQL. Got a nasty error message? Post it on a Forum. Want to suggest a feature? Same thing. What the MemSQL Forums Do The ... Read More

Feed: Recent Announcements. AWS Glue automates much of the effort to build, maintain, and run extract, transform, and load (ETL) jobs. AWS Glue crawls your data sources, identifies data formats, and suggests schemas and transformations. AWS Glue automatically generates the code to execute your data transformations and loading processes. AWS Glue is serverless, so there is no infrastructure to provision or manage. AWS Glue runs your ETL jobs on a fully managed Apache Spark environment.  ... Read More