Tag: Spark

Feed: Recent Announcements. Amazon FinSpace managed Spark clusters now support Apache Spark 3.1.2. Apache Spark 3 has query optimization features like dynamic partition pruning to optimize joins such as joining a large fact table of trades with a smaller dimension table of execution centers. It also includes changes to be more compatible with the ANSI SQL standard and features 30 new built-in functions. FinSpace Spark clusters make it simple for analysts to launch, connect, resize and terminate clusters. FinSpace Spark clusters are available in five sizes, so you can select a configuration that is suitable for your workload, and they ... Read More

Feed: Recent Announcements. Amazon EMR integration with AWS Lake Formation allows you to define and enforce database, table, and column-level permissions when Apache Spark users access data in Amazon S3 through the Glue Data Catalog. Previously, with AWS Lake Formation integration is enabled, you were limited to only being able to read data using Spark SQL statements such as SHOW DATABASES and DESCRIBE TABLE. Now, you can also insert data into, or update the Glue Data Catalog tables with these statements: INSERT INTO, INSERT OVERWRITE, and ALTER TABLE. This feature is enabled on Amazon EMR 5.34 in the following AWS ... Read More

Feed: Recent Announcements. Amazon SageMaker Feature Store is announcing a new enhancement, a connector for Apache Spark that makes batch data ingestion easier for customers. Amazon SageMaker Feature Store is a fully managed, purpose-built repository to store, update, retrieve, and share machine learning (ML) model features. There are various ways to ingest data into SageMaker Feature Store including the PutRecord API, SageMaker Python SDK’s FeatureGroup.ingest functionality and SageMaker Processing Job ... Read More

Feed: Neo4j Graph Database Platform. Author: Adam Cowley. Hey everyone! I’m Adam and I work on the education side of the Dev Rel team at Neo4j, mainly focusing on GraphAcademy. I don’t know about you, but I always find that the week the clocks fall back in Europe plays havoc with my schedule, especially with the regular meetings arranged by my colleagues over in the U.S. For the next month, I will be trawling through the archives to bring you some of the more interesting articles written by our fantastic community members. We start off with some big announcements from ... Read More

Feed: Neo4j Graph Database Platform. Author: Enzo. Neo4j recently released the Neo4j Connector for Apache Spark version 4.1, and in this post, I’d like to cover some of the important changes and new functionality. Today, we’re announcing the availability of streaming Spark support, both source and sink, which allows you to produce or consume data to and from Neo4j AuraDB, from any source you can connect to Spark.These features are important to support the growing customer base using Neo4j AuraDB. We’re pleased to announce these and to make sure that customers know that everything described in this post works with ... Read More

Feed: Neo4j Graph Database Platform. Author: Max Andersson. Hello, everyone! As we begin a new quarter – and enter the spookiest month of the year – Twin4j brings you both tricks and treats from the past week in the world of Neo4j. In this issue, we will see Link Prediction pipelines in focus as we release the new version of GDS. Read about the new features in Neo4j GDS 1.7 and learn how link prediction pipelines can be used to discover travel patterns of digital nomads.   Michael Hunger ... Read More

Feed: Featured Blog Posts - Data Science Central. Author: Sara Petrie. Note: This guide applies to running Spark jobs on any platform, including Cloudera platforms; cloud vendor-specific platforms – Amazon EMR, Microsoft HDInsight, Microsoft Synapse, Google DataProc; Databricks, which is on all three major public cloud providers; and Apache Spark on Kubernetes, which runs on nearly all platforms, including on-premises and cloud. Introduction Spark is known for being extremely difficult to debug. But this is not all Spark’s fault. Problems in running a Spark job can be the result of problems with the infrastructure Spark is running on, inappropriate configuration ... Read More

Feed: Cloudera Blog. Author: Gokul Kamaraj. Posted in Technical | June 10, 2021 4 min read Apache Spark is a very popular analytics engine used for large-scale data processing. It is widely used for many big data applications and use cases. CDP Operational Database Experience Experience (COD) is a CDP Public Cloud service that lets you create and manage operational database instances and it is powered by Apache HBase and Apache Phoenix.  To know more about Apache Spark in CDP and CDP Operational Database Experience, see Apache Spark Overview and CDP Operational Database Experience Overview. Apache Spark enables you to ... Read More

Feed: Recent Announcements. Amazon EMR 6.4 release version now supports Apache Spark 3.1.2 and provides runtime improvements with Amazon EMR Runtime for Apache Spark. Amazon EMR 6.4 provides Presto runtime improvements for PrestoDB 0.254, and runtime improvements for Apache Hive 3.1.2 when you use AWS Glue Data Catalog for your metastore. Amazon EMR 6.4 supports Apache Hudi 0.8.0, Trino 359, PrestoDB 0.254, Apache HBase 2.4.4, Apache Phoenix 5.1.2, Apache Flink 1.13.1, Apache Livy 0.7.1, JupyterHub 1.4.1, Apache Zookeeper 3.5.7 and Apache MXNet 1.8.0. Please see our release guide to learn more. Starting Amazon EMR release versions 5.30 and 6.1 and later, ... Read More

Feed: CIO. Author: . Data engineering, as a separate category of expertise in the world of data science, did not occur in a vacuum. The role of the data engineer originated and evolved as the number of data sources and data products ballooned over the years. Therefore, the role and function of a data engineer is closely associated with a variety of different data-processing platforms such as Apache Hadoop, Apache Spark, and a huge number of specialized tools. In this article, you will find out why Spark should be considered the data engineer's best friend. Spark is the ultimate toolkitData engineers ... Read More