Troubleshooting MSK As Source - Amazon Kinesis Data Firehose
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting MSK As Source

This section describes common troubleshooting steps while using MSK As Source


For troubleshooting processing, transformation or S3 delivery issues, please refer the earlier sections

Hose creation fails

Check the following if your hose with MSK As Source is failing creation

Hose Suspended

Check the following if your hose is in SUSPENDED state

  • Check that the source MSK cluster is in Active state.

  • Check that the source topic exists. In case the topic was deleted and re-created, you will have to delete and re-create the Firehose delivery stream as well.

Hose Backpresurred

The value of DataReadFromSource.Backpressured will be 1 when BytesPerSecondLimit per partition is exceeded or that the normal flow of delivery is slow or stopped.

  • If you are hitting BytesPerSecondLimit please check DataReadFromSource.Bytes metric and request a limit increase.

  • Check the CloudWatch logs, destination metrics, Data Transformation metrics and Format Conversion metrics to identify the bottlenecks.

Incorrect Data Freshness

Data freshness seems incorrect

  • Firehose calculates the data freshness based on the timestamp of the consumed record. To ensure that this timestamp is correctly recorded when the producer record is persisted in the Kafka's broker logs, set the Kafka topic timestamp type configuration to be message.timestamp.type=LogAppendTime.