Writing to Kinesis Data Firehose Using Amazon MSK - Amazon Kinesis Data Firehose
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Writing to Kinesis Data Firehose Using Amazon MSK

You can configure Amazon MSK to send information to a Kinesis Data Firehose delivery stream.

  1. Sign in to the Amazon Web Services Management Console and open the Kinesis Data Firehose console at https://console.amazonaws.cn/firehose/.

  2. Choose Create Delivery Stream.

    In the Choose source and destination section of the page, provide values for the following fields:

    Source

    Choose Amazon MSK to configure a Kinesis Data Firehose delivery stream that uses Amazon MSK as a data source. You can choose between MSK provisioned and MSK-Serverless clusters. You can then use Kinesis Data Firehose to read data easily from a specific Amazon MSK cluster and topic and load it into the specified S3 destination.

    Destination

    Choose Amazon S3 as the destination for your Kinesis Data Firehose delivery stream.

    In the Source settings section of the page, provide values for the following fields:

    Amazon MSK cluster connectivity

    Choose either the Private bootstrap brokers (recommended) or Public bootstrap brokers option based on your cluster configuration. Bootstrap brokers is what Apache Kafka client uses as a starting point to connect to the cluster. Public bootstrap brokers are intended for public access from outside of Amazon, while private bootstrap brokers are intended for access from within Amazon. For more information about Amazon MSK, see Amazon Managed Streaming for Apache Kafka.

    To connect to a provisioned or serverless Amazon MSK cluster through private bootstrap brokers, the cluster must meet all of the following requirements.

    • The cluster must be active.

    • The cluster must have IAM as one of its access control methods.

    • Multi-VPC private connectivity must be enabled for the IAM access control method.

    • You must add to this cluster a resource-based policy which grants Kinesis Data Firehose service principal the permission to invoke the Amazon MSK CreateVpcConnection API.

    To connect to a provisioned Amazon MSK cluster through public bootstrap brokers, the cluster must meet all of the following requirements.

    • The cluster must be active.

    • The cluster must have IAM as one of its access control methods.

    • The cluster must be public-accessible.

    Amazon MSK cluster

    For the same account scenario, specify the ARN of the Amazon MSK cluster from where your Kinesis Data Firehose delivery stream will read data.

    For a cross-account scenario, see Cross-Account Delivery from Amazon MSK.

    Topic

    Specify the Apache Kafka topic from which you want your delivery stream to ingest data. Once the delivery stream is created, you cannot update this topic.

    In the Delivery stream name section of the page, provide values for the following fields:

    Delivery stream name

    Specify the name for your delivery stream.

  3. Next, you can complete the optional step of configuring record transformation and record format conversion. For more information, see Record Transformation and Format Conversion.