Amazon Data Firehose was previously known as Amazon Kinesis Data Firehose
Amazon Data Firehose Data Delivery
After data is sent to your Firehose stream, it is automatically delivered to the destination you choose.
Important
If you use the Kinesis Producer Library (KPL) to write data to a Kinesis data stream, you can use aggregation to combine the records that you write to that Kinesis data stream. If you then use that data stream as a source for your Firehose stream, Amazon Data Firehose de-aggregates the records before it delivers them to the destination. If you configure your Firehose stream to transform the data, Amazon Data Firehose de-aggregates the records before it delivers them to Amazon Lambda. For more information, see Developing Amazon Kinesis Data Streams Producers Using the Kinesis Producer Library and Aggregation in the Amazon Kinesis Data Streams Developer Guide.
Topics
- Data Delivery Format
- Data Delivery Frequency
- Data Delivery Failure Handling
- Amazon S3 Object Name Format
- Index Rotation for the OpenSearch Service Destination
- Delivery Across Amazon Accounts and Across Amazon Regions for HTTP Endpoint Destinations
- Duplicated Records
- How to Pause and Resume a Firehose delivery stream
Data Delivery Format
For data delivery to Amazon Simple Storage Service (Amazon S3), Firehose concatenates multiple incoming records based on the buffering configuration of your delivery stream. It then delivers the records to Amazon S3 as an Amazon S3 object. By default, Firehose concatenates data without any delimiters. If you want to have new line delimiters between records, you can add new line delimiters by enabling the feature in the Firehose console configuration or API parameter.
For data delivery to Amazon Redshift, Firehose first delivers incoming data to your S3 bucket in the format described earlier. Firehose then issues an Amazon Redshift COPY command to load the data from your S3 bucket to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup. Ensure that after Amazon Data Firehose concatenates multiple incoming records to an Amazon S3 object, the Amazon S3 object can be copied to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup. For more information, see Amazon Redshift COPY Command Data Format Parameters.
For data delivery to OpenSearch Service and OpenSearch Serverless, Amazon Data Firehose buffers
incoming records based on the buffering configuration of your Firehose stream. It then
generates an OpenSearch Service or OpenSearch Serverless bulk request to index multiple
records to your OpenSearch Service cluster or OpenSearch Serverless collection. Make
sure that your record is UTF-8 encoded and flattened to a single-line JSON object before
you send it to Amazon Data Firehose. Also, the rest.action.multi.allow_explicit_index
option for your OpenSearch Service cluster must be set to true (default) to take bulk
requests with an explicit index that is set per record. For more information, see OpenSearch Service Configure Advanced Options in the
Amazon OpenSearch Service Developer Guide.
For data delivery to Splunk, Amazon Data Firehose concatenates the bytes that you send. If you want delimiters in your data, such as a new line character, you must insert them yourself. Make sure that Splunk is configured to parse any such delimiters.
When delivering data to an HTTP endpoint owned by a supported third-party service provider, you can use the integrated Amazon Lambda service to create a function to transform the incoming record(s) to the format that matches the format the service provider's integration is expecting. Contact the third-party service provider whose HTTP endpoint you've chosen for your destination to learn more about their accepted record format.
For data delivery to Snowflake, Amazon Data Firehose internally buffers data for one second and uses Snowflake streaming API operations to insert data to Snowflake. By default, records that you insert are flushed and committed to the Snowflake table every second. After you make the insert call, Firehose emits a CloudWatch metric that measures how long it took for the data to be committed to Snowflake. Firehose currently supports only single JSON item as record payload and doesn’t support JSON arrays. Make sure that your input payload is a valid JSON object and is well formed without any extra double quotes, quotes, or escape characters.
Data Delivery Frequency
Each Firehose destination has its own data delivery frequency. For more information, see Buffering hints.
Data Delivery Failure Handling
Each Amazon Data Firehose destination has its own data delivery failure handling.
- Amazon S3
-
Data delivery to your S3 bucket might fail for various reasons. For example, the bucket might not exist anymore, the IAM role that Amazon Data Firehose assumes might not have access to the bucket, the network failed, or similar events. Under these conditions, Amazon Data Firehose keeps retrying for up to 24 hours until the delivery succeeds. The maximum data storage time of Amazon Data Firehose is 24 hours. If data delivery fails for more than 24 hours, your data is lost.
- Amazon Redshift
-
For an Amazon Redshift destination, you can specify a retry duration (0–7200 seconds) when creating a Firehose stream.
Data delivery to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup might fail for several reasons. For example, you might have an incorrect cluster configuration of your Firehose stream, a cluster or workgroup under maintenance, or a network failure. Under these conditions, Amazon Data Firehose retries for the specified time duration and skips that particular batch of Amazon S3 objects. The skipped objects' information is delivered to your S3 bucket as a manifest file in the
errors/folder, which you can use for manual backfill. For information about how to COPY data manually with manifest files, see Using a Manifest to Specify Data Files. - Amazon OpenSearch Service and OpenSearch Serverless
-
For the OpenSearch Service and OpenSearch Serverless destination, you can specify a retry duration (0–7200 seconds) when creating a delivery stream.
Data delivery to your OpenSearch Service cluster or OpenSearch Serverless collection might fail for several reasons. For example, you might have an incorrect OpenSearch Service cluster or OpenSearch Serverless collection configuration of your Firehose stream, an OpenSearch Service cluster or OpenSearch Serverless collection under maintenance, a network failure, or similar events. Under these conditions, Amazon Data Firehose retries for the specified time duration and then skips that particular index request. The skipped documents are delivered to your S3 bucket in the
AmazonOpenSearchService_failed/folder, which you can use for manual backfill.For OpenSearch Service, each document has the following JSON format:
{ "attemptsMade": "(number of index requests attempted)", "arrivalTimestamp": "(the time when the document was received by Firehose)", "errorCode": "(http error code returned by OpenSearch Service)", "errorMessage": "(error message returned by OpenSearch Service)", "attemptEndingTimestamp": "(the time when Firehose stopped attempting index request)", "esDocumentId": "(intended OpenSearch Service document ID)", "esIndexName": "(intended OpenSearch Service index name)", "esTypeName": "(intended OpenSearch Service type name)", "rawData": "(base64-encoded document data)" }For OpenSearch Serverless, each document has the following JSON format:
{ "attemptsMade": "(number of index requests attempted)", "arrivalTimestamp": "(the time when the document was received by Firehose)", "errorCode": "(http error code returned by OpenSearch Serverless)", "errorMessage": "(error message returned by OpenSearch Serverless)", "attemptEndingTimestamp": "(the time when Firehose stopped attempting index request)", "osDocumentId": "(intended OpenSearch Serverless document ID)", "osIndexName": "(intended OpenSearch Serverless index name)", "rawData": "(base64-encoded document data)" } - Splunk
-
When Amazon Data Firehose sends data to Splunk, it waits for an acknowledgment from Splunk. If an error occurs, or the acknowledgment doesn’t arrive within the acknowledgment timeout period, Amazon Data Firehose starts the retry duration counter. It keeps retrying until the retry duration expires. After that, Amazon Data Firehose considers it a data delivery failure and backs up the data to your Amazon S3 bucket.
Every time Amazon Data Firehose sends data to Splunk, whether it's the initial attempt or a retry, it restarts the acknowledgement timeout counter. It then waits for an acknowledgement to arrive from Splunk. Even if the retry duration expires, Amazon Data Firehose still waits for the acknowledgment until it receives it or the acknowledgement timeout is reached. If the acknowledgment times out, Amazon Data Firehose checks to determine whether there's time left in the retry counter. If there is time left, it retries again and repeats the logic until it receives an acknowledgment or determines that the retry time has expired.
A failure to receive an acknowledgement isn't the only type of data delivery error that can occur. For information about the other types of data delivery errors, see Splunk Data Delivery Errors. Any data delivery error triggers the retry logic if your retry duration is greater than 0.
The following is an example error record.
{ "attemptsMade": 0, "arrivalTimestamp": 1506035354675, "errorCode": "Splunk.AckTimeout", "errorMessage": "Did not receive an acknowledgement from HEC before the HEC acknowledgement timeout expired. Despite the acknowledgement timeout, it's possible the data was indexed successfully in Splunk. Amazon Data Firehose backs up in Amazon S3 data for which the acknowledgement timeout expired.", "attemptEndingTimestamp": 13626284715507, "rawData": "MiAyNTE2MjAyNzIyMDkgZW5pLTA1ZjMyMmQ1IDIxOC45Mi4xODguMjE0IDE3Mi4xNi4xLjE2NyAyNTIzMyAxNDMzIDYgMSA0MCAxNTA2MDM0NzM0IDE1MDYwMzQ3OTQgUkVKRUNUIE9LCg==", "EventId": "49577193928114147339600778471082492393164139877200035842.0" } - HTTP endpoint destination
-
When Amazon Data Firehose sends data to an HTTP endpoint destination, it waits for a response from this destination. If an error occurs, or the response doesn’t arrive within the response timeout period, Amazon Data Firehose starts the retry duration counter. It keeps retrying until the retry duration expires. After that, Amazon Data Firehose considers it a data delivery failure and backs up the data to your Amazon S3 bucket.
Every time Amazon Data Firehose sends data to an HTTP endpoint destination, whether it's the initial attempt or a retry, it restarts the response timeout counter. It then waits for a response to arrive from the HTTP endpoint destination. Even if the retry duration expires, Amazon Data Firehose still waits for the response until it receives it or the response timeout is reached. If the response times out, Amazon Data Firehose checks to determine whether there's time left in the retry counter. If there is time left, it retries again and repeats the logic until it receives a response or determines that the retry time has expired.
A failure to receive a response isn't the only type of data delivery error that can occur. For information about the other types of data delivery errors, see HTTP Endpoint Data Delivery Errors
The following is an example error record.
{ "attemptsMade":5, "arrivalTimestamp":1594265943615, "errorCode":"HttpEndpoint.DestinationException", "errorMessage":"Received the following response from the endpoint destination. {"requestId": "109777ac-8f9b-4082-8e8d-b4f12b5fc17b", "timestamp": 1594266081268, "errorMessage": "Unauthorized"}", "attemptEndingTimestamp":1594266081318, "rawData":"c2FtcGxlIHJhdyBkYXRh", "subsequenceNumber":0, "dataId":"49607357361271740811418664280693044274821622880012337186.0" } - Snowflake destination
-
For Snowflake destination, when you create a Firehose stream, you can specify an optional retry duration (0-7200 seconds). The default value for retry duration is 60 seconds.
Data delivery to your Snowflake table might fail for several reasons like an incorrect Snowflake destination configuration, Snowflake outage, a network failure, etc. The retry policy doesn’t apply to non-retriable errors. For example, if Snowflake rejects your JSON payload because it had an extra column that's missing in the table, Firehose doesn’t attempt to deliver it again. Instead, it creates a back up for all the insert failures due to JSON payload issues to your S3 error bucket.
Similarly, if delivery fails due to an incorrect role, table, or database, Firehose doesn’t retry and writes the data to your S3 bucket. Retry duration only applies to failure due to a Snowflake service issue, transient network glitches, etc. Under these conditions, Firehose retries for the specified time duration before delivering them to S3. The failed records are delivered in snowflake-failed/ folder, which you can use for manual backfill.
The following is an example JSON for each record that you deliver to S3.
{ "attemptsMade": 3, "arrivalTimestamp": 1594265943615, "errorCode": "Snowflake.InvalidColumns", "errorMessage": "Snowpipe Streaming does not support columns of type AUTOINCREMENT, IDENTITY, GEO, or columns with a default value or collation", "attemptEndingTimestamp": 1712937865543, "rawData": "c2FtcGxlIHJhdyBkYXRh" }
Amazon S3 Object Name Format
When Firehose delivers data to Amazon S3, S3 object key name follows the format <evaluated prefix><suffix>, where the suffix has the format <delivery stream name>-<delivery stream version>-<year>-<month>-<day>-<hour>-<minute>-<second>-<uuid><file extension> <delivery stream version> begins with 1 and increases by 1 for every configuration change of the Firehose delivery stream. You can change delivery stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption). You can do so by using the Firehose console or the UpdateDestination API operation.
For <evaluated prefix>, Firehose adds a default time prefix in the format YYYY/MM/dd/HH.
This prefix creates a logical hierarchy in the bucket, where each forward slash (/) creates a level in the hierarchy.
You can modify this structure by specifying a custom prefix that includes expressions that are evaluated at runtime.
For information about how to specify a custom prefix, see Custom Prefixes for Amazon Simple Storage Service Objects.
By default, the time zone used for time prefix and suffix is in UTC, but you can change it to a time zone that you prefer. For example, to use Japan Standard Time instead of UTC, you can configure the time zone to Asia/Tokyo in the Amazon Web Services Management Console or in API parameter setting (CustomTimeZone). The following list contains time zones that Firehose supports for S3 prefix configuration.
You cannot change the suffix field except <file extension>. When you enable data format conversion or compression, Firehose will append a file extension based on the configuration. The following table explains the default file extension appended by Firehose:
| Configuration | File extension |
|---|---|
| Data Format Conversion: Parquet | .parquet |
| Data Format Conversion: ORC | .orc |
| Compression: Gzip | .gz |
| Compression: Zip | .zip |
| Compression: Snappy | .snappy |
| Compression: Hadoop-Snappy | .hsnappy |
You can also specify a file extension that you prefer in the Firehose console or API. File extension must start with a period (.) and can contain allowed characters: 0-9a-z!-_.*‘(). File extension cannot exceed 128 characters.
Note
When you specify a file extension, it will override the default file extension that Firehose adds when data format conversion or compression is enabled.
Index Rotation for the OpenSearch Service Destination
For the OpenSearch Service destination, you can specify a time-based index rotation option from one of the following five options: NoRotation, OneHour, OneDay, OneWeek, or OneMonth.
Depending on the rotation option you choose, Amazon Data Firehose appends a portion of the UTC
arrival timestamp to your specified index name. It rotates the appended timestamp
accordingly. The following example shows the resulting index name in OpenSearch Service
for each index rotation option, where the specified index name is
myindex and the arrival timestamp is
2016-02-25T13:00:00Z.
| RotationPeriod | IndexName |
|---|---|
NoRotation |
myindex |
OneHour |
myindex-2016-02-25-13 |
OneDay |
myindex-2016-02-25 |
OneWeek |
myindex-2016-w08 |
OneMonth |
myindex-2016-02 |
Note
With the OneWeek option, Data Firehose auto-create indexes using the
format of <YEAR>-w<WEEK NUMBER> (for example, 2020-w33),
where the week number is calculated using UTC time and according to the following US
conventions:
-
A week starts on Sunday
-
The first week of the year is the first week that contains a Saturday in this year
Delivery Across Amazon Accounts and Across Amazon Regions for HTTP Endpoint Destinations
Amazon Data Firehose supports data delivery to HTTP endpoint destinations across Amazon accounts. The Firehose stream and the HTTP endpoint that you choose as your destination can belong to different Amazon accounts.
Amazon Data Firehose also supports data delivery to HTTP endpoint destinations across Amazon regions.
You can deliver data from a Firehose stream in one Amazon region to an HTTP endpoint in
another Amazon region. You can also delivery data from a Firehose stream to an HTTP
endpoint destination outside of Amazon regions, for example to your own on-premises
server by setting the HTTP endpoint URL to your desired destination. For these
scenarios, additional data transfer charges are added to your delivery costs. For more
information, see the Data
Transfer
Duplicated Records
Amazon Data Firehose uses at-least-once semantics for data delivery. In some circumstances, such as when data delivery times out, delivery retries by Amazon Data Firehose might introduce duplicates if the original data-delivery request eventually goes through. This applies to all destination types that Amazon Data Firehose supports.
How to Pause and Resume a Firehose delivery stream
After you setup a delivery stream in Firehose, data available in the stream source is continuously delivered to the destination. If you encounter situations where your stream destination is temporarily unavailable (for example, during planned maintenance operations), you may want to temporarily pause data delivery, and resume when the destination becomes available again. The following sections show how you can accomplish this:
Important
When you use the approach described below to pause and resume a stream, after you resume the stream, you will see that few records get delivered to the error bucket in Amazon S3 while the rest of the stream continues to get delivered to the destination. This is a known limitation of the approach, and it occurs because a small number of records that could not be previously delivered to the destination after multiple retries are tracked as failed.
Understanding how Firehose handles delivery failures
When you setup a delivery stream in Firehose, for many destinations such as OpenSearch, Splunk, and HTTP endpoints, you also setup an S3 bucket where data that fails to be delivered can be backed up. For more information about how Firehose backs up data in case of failed deliveries, see Data Delivery Failure Handling. For more information about how to grant access to S3 buckets where data that fails to be delivered can be backed up, see Grant Firehose Access to an Amazon S3 Destination. When Firehose (a) fails to deliver data to the stream destination, and (b) fails to write data to the backup S3 bucket for failed deliveries, it effectively pauses stream delivery until such time that data can either be delivered to the destination or written to the backup S3 location.
Pausing a Firehose delivery stream
To pause stream delivery in Firehose, first remove permissions for Firehose to write to the S3 backup location for failed deliveries. For example, if you want to pause the delivery stream with an OpenSearch destination, you can do this by updating permissions. For more information, see Grant Firehose Access to a Public OpenSearch Service Destination.
Remove the "Effect": "Allow" permission for the action s3:PutObject, and explicitly add a statement that applies Effect": "Deny" permission on the action
s3:PutObject for the S3 bucket used for backing up failed deliveries. Next, turn off the stream destination
(for example, turning off the destination OpenSearch domain), or remove permissions for Firehose to write to the destination.
To update permissions for other destinations, check the section for your destination in
Controlling Access with Amazon Data Firehose.
After you complete these two actions, Firehose will stop delivering streams, and you can monitor this using
CloudWatch metrics for Firehose.
Important
When you pause stream delivery in Firehose, you need to ensure that the source of the stream (for example, in Kinesis Data Streams or in Managed Service for Kafka) is configured to retain data until stream delivery is resumed and the data gets delivered to the destination. If the source is DirectPUT, Firehose will retain data for 24 hours. Data loss could happen if you do not resume the stream and deliver the data before the expiration of data retention period.
Resuming a Firehose delivery stream
To resume delivery, first revert the change made earlier to the stream destination by turning on the destination and ensuring that
Firehose has permissions to deliver the stream to the destination. Next, revert the changes made earlier to permissions applied
to the S3 bucket for backing up failed deliveries. That is, apply "Effect": "Allow" permission for the action s3:PutObject,
and remove "Effect": "Deny" permission on the action s3:PutObject for the S3 bucket used for backing up failed deliveries.
Finally, monitor using CloudWatch metrics for Firehose to confirm that the stream is being delivered to the destination.
To view and troubleshoot errors, use Amazon CloudWatch Logs monitoring for Firehose.