Monitoring Kinesis Data Firehose Using CloudWatch Metrics
Important
Be sure to enable alarms on all CloudWatch metrics that belong to your destination in order to identify errors in timely manner.
Kinesis Data Firehose integrates with Amazon CloudWatch metrics so that you can collect, view, and analyze CloudWatch
metrics for your Kinesis Data Firehose delivery streams. For example, you can monitor the
IncomingBytes and IncomingRecords metrics to keep track of
data ingested into Kinesis Data Firehose from data producers.
Kinesis Data Firehose collects and publishes CloudWatch metrics every minute. However, if bursts of incoming data occur only for a few seconds, they may not be fully captured or visible in the one-minute metrics. This is because CloudWatch metrics are aggregated from Kinesis Data Firehose over one-minute intervals.
The metrics collected for Kinesis Data Firehose delivery streams are free of charge. For information about Kinesis agent metrics, see Monitoring Kinesis Agent Health.
Topics
- Dynamic Partitioning CloudWatch Metrics
- Data Delivery CloudWatch Metrics
- Data Ingestion Metrics
- API-Level CloudWatch Metrics
- Data Transformation CloudWatch Metrics
- Format Conversion CloudWatch Metrics
- Server-Side Encryption (SSE) CloudWatch Metrics
- Dimensions for Kinesis Data Firehose
- Kinesis Data Firehose Usage Metrics
Dynamic Partitioning CloudWatch Metrics
If dynamic partitioning
| Metric | Description |
|---|---|
ActivePartitionsLimit |
The maximum number of active partitions that a delivery stream processes before sending data to the error bucket. Units: Count |
PartitionCount |
The number of partitions that are being processed, in other words, the active partition count. This number varies between 1 and the partition count limit of 500 (default). Units: Count |
PartitionCountExceeded |
This metric indicates if you are exceeding the partition count limit. It emits 1 or 0 based on whether limit is breached or not. |
JQProcessing.Duration |
Returns the amount of time it took to execute JQ expression in the JQ Lambda function. Units: Milliseconds |
PerPartitionThroughput |
Indicates the throughtput that is being processed per partition. This metric enables you to monitor the per partition throughput. Units: StandardUnit.BytesSecond |
DeliveryToS3.ObjectCount |
Indicates the number of objects that are being delivered to your S3 bucket. Units: Count |
Data Delivery CloudWatch Metrics
The Amazon/Firehose namespace includes the following service-level
metrics. If you see small drops in the average for BackupToS3.Success,
DeliveryToS3.Success, DeliveryToSplunk.Success,
DeliveryToAmazonOpenSearchService.Success, or
DeliveryToRedshift.Success, that doesn't indicate that there's data
loss. Kinesis Data Firehose retries delivery errors and doesn't move forward until the records are
successfully delivered either to the configured destination or to the backup S3
bucket.
Delivery to OpenSearch Service
| Metric | Description |
|---|---|
DeliveryToAmazonOpenSearchService.Bytes |
The number of bytes indexed to OpenSearch Service over the specified time period. Units: Bytes |
DeliveryToAmazonOpenSearchService.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to OpenSearch Service. Units: Seconds |
DeliveryToAmazonOpenSearchService.Records |
The number of records indexed to OpenSearch Service over the specified time period. Units: Count |
DeliveryToAmazonOpenSearchService.Success |
The sum of the successfully indexed records over the sum of records that were attempted. |
DeliveryToS3.Bytes |
The number of bytes delivered to Amazon S3 over the specified time period. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Count |
DeliveryToS3.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Seconds |
DeliveryToS3.Records |
The number of records delivered to Amazon S3 over the specified time period. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Count |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. Kinesis Data Firehose always emits this metric regardless of whether backup is enabled for failed documents only or for all documents. |
DeliveryToAmazonOpenSearchService.AuthFailure |
Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates authentication failure. |
DeliveryToAmazonOpenSearchService.DeliveryRejected |
Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there's a delivery failure. |
Delivery to OpenSearch Serverless
| Metric | Description |
|---|---|
DeliveryToAmazonOpenSearchServerless.Bytes |
The number of bytes indexed to OpenSearch Serverless over the specified time period. Units: Bytes |
DeliveryToAmazonOpenSearchServerless.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to OpenSearch Serverless. Units: Seconds |
DeliveryToAmazonOpenSearchServerless.Records |
The number of records indexed to OpenSearch Serverless over the specified time period. Units: Count |
DeliveryToAmazonOpenSearchServerless.Success |
The sum of the successfully indexed records over the sum of records that were attempted. |
DeliveryToS3.Bytes |
The number of bytes delivered to Amazon S3 over the specified time period. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Count |
DeliveryToS3.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Seconds |
DeliveryToS3.Records |
The number of records delivered to Amazon S3 over the specified time period. Kinesis Data Firehose emits this metric only when you enable backup for all documents. Units: Count |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. Kinesis Data Firehose always emits this metric regardless of whether backup is enabled for failed documents only or for all documents. |
DeliveryToAmazonOpenSearchServerless.AuthFailure |
Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there is an authentication failure. |
DeliveryToAmazonOpenSearchServerless.DeliveryRejected |
Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there is a delivery failure. |
Delivery to Amazon Redshift
| Metric | Description |
|---|---|
DeliveryToRedshift.Bytes |
The number of bytes copied to Amazon Redshift over the specified time period. Units: Count |
DeliveryToRedshift.Records |
The number of records copied to Amazon Redshift over the specified time period. Units: Count |
DeliveryToRedshift.Success |
The sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands. |
DeliveryToS3.Bytes |
The number of bytes delivered to Amazon S3 over the specified time period. Units: Bytes |
DeliveryToS3.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Units: Seconds |
DeliveryToS3.Records |
The number of records delivered to Amazon S3 over the specified time period. Units: Count |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. |
BackupToS3.Bytes |
The number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled. Units: Count |
BackupToS3.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled. Units: Seconds |
BackupToS3.Records |
The number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled. Units: Count |
BackupToS3.Success |
Sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when backup to Amazon S3 is enabled. |
Delivery to Amazon S3
The metrics in the following table are related to delivery to Amazon S3 when it is the main destination of the delivery stream.
| Metric | Description |
|---|---|
DeliveryToS3.Bytes |
The number of bytes delivered to Amazon S3 over the specified time period. Units: Bytes |
DeliveryToS3.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Units: Seconds |
DeliveryToS3.Records |
The number of records delivered to Amazon S3 over the specified time period. Units: Count |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. |
BackupToS3.Bytes |
The number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup is enabled (which is only possible when data transformation is also enabled). Units: Count |
BackupToS3.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when backup is enabled (which is only possible when data transformation is also enabled). Units: Seconds |
BackupToS3.Records |
The number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when backup is enabled (which is only possible when data transformation is also enabled). Units: Count |
BackupToS3.Success |
Sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when backup is enabled (which is only possible when data transformation is also enabled). |
Delivery to Splunk
| Metric | Description |
|---|---|
DeliveryToSplunk.Bytes |
The number of bytes delivered to Splunk over the specified time period. Units: Bytes |
DeliveryToSplunk.DataAckLatency |
The approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data. The increasing or decreasing trend for this metric is more useful than the absolute approximate value. Increasing trends can indicate slower indexing and acknowledgement rates from Splunk indexers. Units: Seconds |
DeliveryToSplunk.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to Splunk. Units: Seconds |
DeliveryToSplunk.Records |
The number of records delivered to Splunk over the specified time period. Units: Count |
DeliveryToSplunk.Success |
The sum of the successfully indexed records over the sum of records that were attempted. |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. This metric is emitted when backup to Amazon S3 is enabled. |
BackupToS3.Bytes |
The number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when the delivery stream is configured to back up all documents. Units: Count |
BackupToS3.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when the delivery stream is configured to back up all documents. Units: Seconds |
BackupToS3.Records |
The number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when the delivery stream is configured to back up all documents. Units: Count |
BackupToS3.Success |
Sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when the delivery stream is configured to back up all documents. |
Delivery to HTTP Endpoints
| Metric | Description |
|---|---|
DeliveryToHttpEndpoint.Bytes |
The number of bytes delivered successfully to the HTTP endpoint. Units: Bytes |
DeliveryToHttpEndpoint.Records |
The number of records delivered successfully to the HTTP endpoint. Units: Counts |
DeliveryToHttpEndpoint.DataFreshness |
Age of the oldest record in Kinesis Data Firehose. Units: Seconds |
DeliveryToHttpEndpoint.Success |
The sum of all successful data delivery requests to the HTTP endpoint Units: Count |
DeliveryToHttpEndpoint.ProcessedBytes |
The number of attempted processed bytes, including retries. |
DeliveryToHttpEndpoint.ProcessedRecords |
The number of attempted records including retries. |
Data Ingestion Metrics
Data Ingestion Through Kinesis Data Streams
| Metric | Description |
|---|---|
DataReadFromKinesisStream.Bytes |
When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. This number includes rereads due to failovers. Units: Bytes |
DataReadFromKinesisStream.Records |
When the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream. This number includes rereads due to failovers. Units: Count |
ThrottledDescribeStream |
The total number of times the Units: Count |
ThrottledGetRecords |
The total number of times the Units: Count |
ThrottledGetShardIterator |
The total number of times the
Units: Count |
Data Ingestion Through Direct PUT
| Metric | Description |
|---|---|
BackupToS3.Bytes |
The number of bytes delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations. Units: Bytes |
BackupToS3.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations. Units: Seconds |
BackupToS3.Records |
The number of records delivered to Amazon S3 for backup over the specified time period. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations. Units: Count |
BackupToS3.Success |
Sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations. |
BytesPerSecondLimit |
The current maximum number of bytes per second that a
delivery stream can ingest before throttling. To request an
increase to this limit, go to the Amazon
Support Center |
DataReadFromKinesisStream.Bytes |
When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. This number includes rereads due to failovers. Units: Bytes |
DataReadFromKinesisStream.Records |
When the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream. This number includes rereads due to failovers. Units: Count |
DeliveryToAmazonOpenSearchService.Bytes |
The number of bytes indexed to OpenSearch Service over the specified time period. Units: Bytes |
DeliveryToAmazonOpenSearchService.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to OpenSearch Service. Units: Seconds |
DeliveryToAmazonOpenSearchService.Records |
The number of records indexed to OpenSearch Service over the specified time period. Units: Count |
DeliveryToAmazonOpenSearchService.Success |
The sum of the successfully indexed records over the sum of records that were attempted. |
DeliveryToRedshift.Bytes |
The number of bytes copied to Amazon Redshift over the specified time period. Units: Bytes |
DeliveryToRedshift.Records |
The number of records copied to Amazon Redshift over the specified time period. Units: Count |
DeliveryToRedshift.Success |
The sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands. |
DeliveryToS3.Bytes |
The number of bytes delivered to Amazon S3 over the specified time period. Units: Bytes |
DeliveryToS3.DataFreshness |
The age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Units: Seconds |
DeliveryToS3.Records |
The number of records delivered to Amazon S3 over the specified time period. Units: Count |
DeliveryToS3.Success |
The sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. |
DeliveryToSplunk.Bytes |
The number of bytes delivered to Splunk over the specified time period. Units: Bytes |
DeliveryToSplunk.DataAckLatency |
The approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data. The increasing or decreasing trend for this metric is more useful than the absolute approximate value. Increasing trends can indicate slower indexing and acknowledgement rates from Splunk indexers. Units: Seconds |
DeliveryToSplunk.DataFreshness |
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to Splunk. Units: Seconds |
DeliveryToSplunk.Records |
The number of records delivered to Splunk over the specified time period. Units: Count |
DeliveryToSplunk.Success |
The sum of the successfully indexed records over the sum of records that were attempted. |
IncomingBytes |
The number of bytes ingested successfully into the delivery stream over the specified time period. Data ingestion could be throttled when it exceeds one of the delivery stream limits.
Throttled data will not be counted for Units: Bytes |
IncomingPutRequests |
The number of successful PutRecord and PutRecordBatch requests over a specified period of time. Units: Count |
IncomingRecords |
The number of records ingested successfully into the delivery stream over the specified time period. Data ingestion could be throttled when it exceeds one of the delivery stream limits.
Throttled data will not be counted for Units: Count |
KinesisMillisBehindLatest |
When the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data stream. Units: Millisecond |
RecordsPerSecondLimit |
The current maximum number of records per second that a delivery stream can ingest before throttling. Units: Count |
ThrottledRecords |
The number of records that were throttled because data ingestion exceeded one of the delivery stream limits. Units: Count |
Data Ingestion From MSK
| Metric | Description |
|---|---|
DataReadFromSource.Records
|
The number of records read from the source Kafka Topic. Units: Count |
DataReadFromSource.Bytes
|
The number of bytes read from the source Kafka Topic. Units: Bytes |
SourceThrottled.Delay
|
The amount of time that the source Kafka cluster is delayed in returning the records from the source Kafka Topic. Units: Milliseconds |
BytesPerSecondLimit
|
Current limit of throughput at which Firehose is going to read from each partition of the source Kafka Topic. Units: Bytes/sec |
KafkaOffsetLag
|
The difference between the largest offset of the record that Firehose has read from the source Kafka Topic and the largest offset of the record available from the source Kafka Topic. Units: Count |
FailedValidation.Records
|
The number of records that failed record validation. Units: Count |
FailedValidation.Bytes
|
The number of bytes that failed record validation. Units: Bytes |
DataReadFromSource.Backpressured
|
Indicates that a delivery stream is delayed in reading records from the source partition either because BytesPerSecondLimit per partition has exceeded or that the normal flow of delivery is slow or has stopped Units: Boolean |
API-Level CloudWatch Metrics
The Amazon/Firehose namespace includes the following API-level
metrics.
| Metric | Description |
|---|---|
DescribeDeliveryStream.Latency |
The time taken per Units: Milliseconds |
DescribeDeliveryStream.Requests |
The total number of Units: Count |
ListDeliveryStreams.Latency |
The time taken per Units: Milliseconds |
ListDeliveryStreams.Requests |
The total number of Units: Count |
PutRecord.Bytes |
The number of bytes put to the Kinesis Data Firehose delivery stream using
Units: Bytes |
PutRecord.Latency |
The time taken per Units: Milliseconds |
PutRecord.Requests |
The total number of Units: Count |
PutRecordBatch.Bytes |
The number of bytes put to the Kinesis Data Firehose delivery stream using
Units: Bytes |
PutRecordBatch.Latency |
The time taken per Units: Milliseconds |
PutRecordBatch.Records |
The total number of records from Units: Count |
PutRecordBatch.Requests |
The total number of Units: Count |
PutRequestsPerSecondLimit |
The maximum number of put requests per second that a delivery stream can handle before throttling. This number includes PutRecord and PutRecordBatch requests. Units: Count |
ThrottledDescribeStream |
The total number of times the Units: Count |
ThrottledGetRecords |
The total number of times the Units: Count |
ThrottledGetShardIterator |
The total number of times the Units: Count |
UpdateDeliveryStream.Latency |
The time taken per Units: Milliseconds |
UpdateDeliveryStream.Requests |
The total number of Units: Count |
Data Transformation CloudWatch Metrics
If data transformation with Lambda is enabled, the AWS/Firehose namespace
includes the following metrics.
| Metric | Description |
|---|---|
ExecuteProcessing.Duration |
The time it takes for each Lambda function invocation performed by Kinesis Data Firehose. Units: Milliseconds |
ExecuteProcessing.Success |
The sum of the successful Lambda function invocations over the sum of the total Lambda function invocations. |
SucceedProcessing.Records |
The number of successfully processed records over the specified time period. Units: Count |
SucceedProcessing.Bytes |
The number of successfully processed bytes over the specified time period. Units: Bytes |
Format Conversion CloudWatch Metrics
If format conversion is enabled, the AWS/Firehose namespace
includes the following metrics.
| Metric | Description |
|---|---|
SucceedConversion.Records |
The number of successfully converted records. Units: Count |
SucceedConversion.Bytes |
The size of the successfully converted records. Units: Bytes |
FailedConversion.Records |
The number of records that could not be converted. Units: Count |
FailedConversion.Bytes |
The size of the records that could not be converted. Units: Bytes |
Server-Side Encryption (SSE) CloudWatch Metrics
The Amazon/Firehose namespace includes the following metrics that are
related to SSE.
| Metric | Description |
|---|---|
KMSKeyAccessDenied |
The number of times the service encounters a
Units: Count |
KMSKeyDisabled |
The number of times the service encounters a
Units: Count |
KMSKeyInvalidState |
The number of times the service encounters a
Units: Count |
KMSKeyNotFound |
The number of times the service encounters a
Units: Count |
Dimensions for Kinesis Data Firehose
To filter metrics by delivery stream, use the DeliveryStreamName
dimension.
Kinesis Data Firehose Usage Metrics
You can use CloudWatch usage metrics to provide visibility into your account's usage of resources. Use these metrics to visualize your current service usage on CloudWatch graphs and dashboards.
Service quota usage metrics are in the Amazon/Usage namespace and are collected every minute.
Currently, the only metric name in this namespace that CloudWatch publishes is
ResourceCount. This metric is published with the dimensions
Service, Class, Type, and
Resource.
| Metric | Description |
|---|---|
ResourceCount |
The number of the specified resources running in your account. The resources are defined by the dimensions associated with the metric. The most useful statistic for this metric is MAXIMUM, which represents the maximum number of resources used during the 1-minute period. |
The following dimensions are used to refine the usage metrics that are published by Kinesis Data Firehose.
| Dimension | Description |
|---|---|
Service |
The name of the Amazon service containing the resource. For
Kinesis Data Firehose usage metrics, the value for this dimension is
|
Class |
The class of resource being tracked. Kinesis Data Firehose API usage metrics
use this dimension with a value of |
Type |
The type of resource being tracked. Currently, when the
Service dimension is |
Resource |
The name of the Amazon resource. Currently, when the Service
dimension is |