Troubleshooting MSK As Source
This section describes common troubleshooting steps while using MSK As Source
Note
For troubleshooting processing, transformation or S3 delivery issues, please refer the earlier sections
Hose creation fails
Check the following if your hose with MSK As Source is failing creation
-
Check that the source MSK cluster is in Active state.
-
If you are using Private connectivity, ensure that Private Link on the cluster is turned on
If you are using Public connectivity, ensure that Public access on the cluster is turned on
-
If you are using Private connectivity, make sure that you add a resource based policy that allows Firehose to create Private Link
. Also refer: MSK cross account permissions -
Ensure that the role in source configuration has permission to ingest data from cluster's Topic
-
Ensure that your VPC security groups allow incoming traffic on ports used by the cluster's bootstrap servers
Hose Suspended
Check the following if your hose is in SUSPENDED state
-
Check that the source MSK cluster is in Active state.
-
Check that the source topic exists. In case the topic was deleted and re-created, you will have to delete and re-create the Firehose delivery stream as well.
Hose Backpresurred
The value of DataReadFromSource.Backpressured will be 1 when BytesPerSecondLimit per partition is exceeded or that the normal flow of delivery is slow or stopped.
-
If you are hitting BytesPerSecondLimit please check DataReadFromSource.Bytes metric and request a limit increase.
-
Check the CloudWatch logs, destination metrics, Data Transformation metrics and Format Conversion metrics to identify the bottlenecks.
Incorrect Data Freshness
Data freshness seems incorrect
-
Firehose calculates the data freshness based on the timestamp of the consumed record. To ensure that this timestamp is correctly recorded when the producer record is persisted in the Kafka's broker logs, set the Kafka topic timestamp type configuration to be
message.timestamp.type=LogAppendTime.