Tag: Kafka

Feed: Recent Announcements. AWS Glue can now connect to Apache Kafka using additional client authentication mechanisms. AWS Glue now supports SASL (Simple Authentication and Security Layer) using either SCRAM (Salted Challenge Response Authentication Mechanism) or GSSAPI (Kerberos). AWS Glue supports data streams including Amazon Kinesis and Apache Kafka, applies complex transformations in-flight and loads it into a target data store for Analytics and Machine Learning. With this feature, you can now stream data from Apache Kafka producers that use SASL (SCRAM and GSSAPI) for client authentication. You can choose from these client authentication mechanisms when creating a Kafka connection in ... Read More

Feed: AWS Big Data Blog. You can use Apache Kafka to run your streaming workloads. Kafka provides resiliency to failures and protects your data out of the box by replicating data across the brokers of the cluster. This makes sure that the data in the cluster is durable. You can achieve your durability SLAs by changing the replication factor of the topic. However, streaming data stored in Kafka topics tends to be transient and typically has a retention time of days or weeks. You may want to back up the data stored in your Kafka topic long after its retention ... Read More

Feed: AWS Big Data Blog. Apache Kafka is well known for its performance and tunability to optimize for various use cases. But sometimes it can be challenging to find the right infrastructure configuration that meets your specific performance requirements while minimizing the infrastructure cost. This post explains how the underlying infrastructure affects Apache Kafka performance. We discuss strategies on how to size your clusters to meet your throughput, availability, and latency requirements. Along the way, we answer questions like “when does it make sense to scale up vs. scale out?” We end with guidance on how to continuously verify the ... Read More

Feed: Neo4j Graph Data Platform. Author: Yolande Poirier. Welcome to Neo4j Under The Hood! – a series of short videos, presented by our top engineering leaders. Chris Gioran, our Chief Architect, kicks off this series explaining the power of graph databases and their transformative use cases. He introduces the fundamental concepts and showcases the nuts and bolts of graph technology. We hope you enjoy these videos and encourage you to sign up to get an alert for upcoming episodes ... Read More

Feed: Recent Announcements. AWS Lambda has launched improved auto scaling for Amazon MSK and self-managed Kafka as event sources to improve performance and help lower costs for customers. Lambda starts with one consumer and checks the OffsetLag metric (measure of backlog at source) every minute and scales up or down every 3 minutes. Lambda allows up to one consumer per partition per topic for a Kafka cluster. Previously, Lambda used to start connection with Kafka clusters at 50% of maximum number of consumers, check for OffsetLag, and scale up consumers every 15 minutes ... Read More

Feed: Recent Announcements. Amazon MSK is a fully managed service for Apache Kafka and Kafka Connect that makes it easy for you to build and run applications that use Apache Kafka as a data store. Amazon MSK is fully compatible with Apache Kafka, which enables you to quickly migrate your existing Apache Kafka workloads to Amazon MSK with confidence or build new ones from scratch. With Amazon MSK, you spend more time building innovative applications and less time managing clusters. To learn how to get started, see the Amazon MSK Developer Guide ... Read More

Feed: Recent Announcements. Amazon MSK is a fully managed service for Apache Kafka and Kafka Connect that makes it easy for you to build and run applications that use Apache Kafka as a data store. Amazon MSK is fully compatible with Apache Kafka, which enables you to quickly migrate your existing Apache Kafka workloads to Amazon MSK with confidence or build new ones from scratch. With Amazon MSK, you spend more time building innovative applications and less time managing clusters. To learn how to get started, see the Amazon MSK Developer Guide ... Read More

Feed: Recent Announcements. AWS Lambda now supports Max Batching Window, a new feature that allows developers to fine tune Lambda invocation for cost optimization. This feature gives you additional control on batching behavior when processing data from Amazon MSK, Apache Kafka, Amazon MQ for Apache Active MQ and RabbitMQ as event sources. Lambda reads records from an event source stream at a fixed cadence and invokes a function with a batch of records. Max Batching Window allows you to wait as long as 300s to build a batch before invoking a function. Now, a function is invoked when one of ... Read More

Feed: Planet PostgreSQL. In parts 6 and of the pipeline series we took a different path in the pipe/tunnel and explored PostgreSQL and Apache Superset, mainly from a functional perspective—how can you get JSON data into PostgreSQL from Apache Kafka Connect, and what does it look like in Superset. In part 8, we ran some initial load tests and found out how the capacity of the original Elasticsearch pipeline compared with the PostgreSQL variant. These results were surprising, so in true “Myth Busters” style we had another attempt to make them more comparable (Part 9). In this blog, we sum ... Read More

Feed: Recent Announcements. Amazon MSK is a fully managed service for Apache Kafka and Kafka Connect that makes it easy for you to build and run applications that use Apache Kafka as a data store. Amazon MSK is fully compatible with Apache Kafka, which enables you to quickly migrate your existing Apache Kafka workloads to Amazon MSK with confidence or build new ones from scratch. With Amazon MSK, you can spend more time innovating on applications and less time managing clusters. To learn how to get started, see the Amazon MSK Developer Guide ... Read More