Using Lambda with Apache Kafka - Amazon Lambda
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Using Lambda with Apache Kafka

Lambda supports Apache Kafka as an event source. Apache Kafka is an open-source event streaming platform designed to handle high-throughput, real-time data pipelines and streaming applications. There are two main ways to use Lambda with Apache Kafka:

  • Using Lambda with Amazon MSK – Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully-managed service by Amazon. Amazon MSK helps automate management of your Kafka infrastructure, including provisioning, patching, and scaling.

  • Using Lambda with self-managed Apache Kafka – In Amazon terminology, a self-managed cluster includes non-Amazon hosted Kafka clusters. For example, you can still use Lambda with a Kafka cluster hosted with a non-Amazon cloud provider such as Confluent Cloud or Redpanda.

When deciding between Amazon MSK and self-managed Apache Kafka, consider your operational needs and control requirements. Amazon MSK is a better choice if you want Amazon to quickly help you manage a scalable, production-ready Kafka setup with minimal operational overhead. It simplifies security, monitoring, and high availability, helping you focus on application development rather than infrastructure management. On the other hand, self-managed Apache Kafka is better suited for use cases running on non-Amazon hosted environments, including on-premises clusters.

Topics