Tag: Spark

Feed: What's New. You can now use Python scripts in AWS Glue to run small to medium-sized generic tasks that are often part of an ETL (extract, transform, and load) workflow. Previously, AWS Glue jobs were limited to those that ran in a serverless Apache Spark environment. You can now use Python shell jobs, for example, to submit SQL queries to services such as Amazon Redshift, Amazon Athena, or Amazon EMR, or run machine-learning and scientific analyses ... Read More

Feed: AWS Big Data Blog. In this post, we talk about designing a cloud-native data warehouse as a replacement for our legacy data warehouse built on a relational database.At the beginning of the design process, the simplest solution appeared to be a straightforward lift-and-shift migration from one relational database to another. However, we decided to step back and focus first on what we really needed out of a data warehouse. We started looking at how we could decouple our legacy Oracle database into smaller microservices, using the right tool for the right job. Our process wasn’t just about using the ... Read More

Feed: Microsoft Azure Blog. Author: Rob Caron. Now in preview Azure Monitor logs in Grafana - now in public preview Grafana offers great dashboarding capabilities, rich visualizations, and integrations with over 40 data sources. Grafana integration is now available in preview for Microsoft Azure Monitor logs. This integration is achieved through the new Log Analytics plugin, now available as part of the Azure Monitor data source. If you’re already using Grafana for your dashboards, this new plugin can help you create a single pane of glass for your various monitoring needs. The new plugin enables you to display any data ... Read More

Feed: IBM Big Data & Analytics Hub - All Content; Author: thomas-chu; Organizations are continuing to turn to hybrid data management to establish a solid data foundation for day-to-day operations and more forward-looking capabilities such as machine learning (ML) and artificial intelligence (AI).With THINK 2019 just around the corner, 12 through 15 February, there’s no better time to discover the variety of hybrid data management solutions and strategies, along with how each can help uncover actionable insights. Think 2019 will be full of learning opportunities that range from sessions with IBM experts and clients to hands-on labs, roundtable discussions and ... Read More

Feed: Hortonworks Blog – Hortonworks. Author: Abhas Ricky. The 2018 Hortonworks’ DataWorks Summit events offered me a unique opportunity to talk to customers about the technology zeitgeist. I attended the Hortonworks events in Singapore and Japan, and I came away with a clearer understanding of several emerging technology trends in Asia that are shaping the future of business and fuelling the adoption of hybrid cloud. These events also provided a chance to hear how Fortune 500 executives are creating next-generation data platforms to support future innovations. As-a-Service Models Are Taking Over As I traveled and talked with current and potential ... Read More

Feed: Neo4j Graph Database Platform. Author: Bryce Merkl Sasaki. Editor’s Note: Alastair Green (Neo4j Query Languages Lead) and Martin Junghanns (Neo4j Cypher for Apache Spark project) will be speaking at the Spark + AI Summit in San Francisco this April.Want to Extend the Graph Capabilities of Spark?Let’s get straight to it: The Apache Spark community is joining hands with the growing graph data movement, and we need your help to keep moving forward.Databricks contributor Xiangrui Meng is sponsoring the idea of property graphs built on Spark DataFrames, with Cypher queries, along with graph algorithms from the GraphFrames project. This would ... Read More

Feed: Planet MySQL; Author: Alexander Rubin; PREVIOUS POST Parallel query execution is my favorite, non-existent, feature in MySQL. In all versions of MySQL – at least at the time of writing – when you run a single query it will run in one thread, effectively utilizing one CPU core only. Multiple queries run at the same time will be using different threads and will utilize more than one CPU core. On multi-core machines – which is the majority of the hardware nowadays – and in the cloud, we have multiple cores available for use. With faster disks (i.e. SSD) we can’t utilize ... Read More

Feed: MemSQL Blog. Author: Floyd Smith. What makes MemSQL unique? Its ability to combine the scalability of NoSQL databases and the familiar structure, transaction capabilities, and SQL support of traditional relational databases. MemSQL does all this while offering high performance on ingest, fast query response, and high concurrency – support for many active users at the same time. Because of its SQL support, MemSQL is “plug and play” with existing database systems and with all the people and tools that are part of the SQL ecosystem. The combination of capabilities that are offered in MemSQL are not found together anywhere ... Read More

Feed: Microsoft Azure Blog. Author: Tyler Fox. We recently introduced support for HDInsight in Microsoft Azure CLI as a public preview. With the addition of the new HDInsight command group, you can now utilize all of the features and benefits that come with the familiar cross-platform Azure CLI to manage your HDInsight clusters. Key Features Cluster CRUD: Create, delete, list, resize, and show properties for your HDInsight clusters. Script actions: Execute script actions, list and delete persistent script actions, promote ad-hoc script executions to persistent script actions, and show the execution history of script actions on HDInsight clusters. Operations Management ... Read More

Feed: Planet big data. Author: Rav Ahuja. Still waiting…it’s been over an hour and still nothing. I watch the clock, get some tea, ruminate on the structure of dark matter…. I’m trying to work with course enrollment data in a relatively large database and format it in a nice splashboard, but processing this data takes far too long. Perhaps dark matter is to blame. Let me back up. Last year, I was tinkering with a Jupter notebook to summarize course enrollment and completion stats for some of our database courses. In fact I started with a notebook that someone had ... Read More