Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon EventBridge Pipes

Amazon EventBridge Pipes connects sources to targets. It reduces the need for specialized knowledge and integration code when developing event driven architectures, fostering consistency across your company’s applications. To set up a pipe, you choose the source, add optional filtering, define optional enrichment, and choose the target for the event data.

EventBridge Pipes could be used to create an ecommerce system. Imagine that you have an API that contains customer metadata, such as shipping addresses. The source might be an Amazon SQS order received message. The pipe then sends that data to an Amazon API Gateway API enrichment that returns the customer information for that order. Finally the pipe sends the enriched data to a Amazon Step Functions state machine, which processes the order.

Source

EventBridge Pipes receives event data from a variety of sources, applies optional filters and enrichment to that data, and sends it to a target. If a source enforces order to the events sent to pipes, that order is maintained throughout the entire process to the target. For more information about sources, see Amazon EventBridge Pipes sources.

Filtering

EventBridge Pipes can filter a given source’s events and then process only a subset of those events. For more information about filtering, see Amazon EventBridge Pipes filtering.

Enrichment

With the enrichment step of EventBridge Pipes, you can enhance the data from the source before sending it to the target. For example, you might receive Ticket created events that don’t include the full ticket data. Using enrichment, you can have a Lambda function call the get-ticket API for the full ticket details. Pipes can then send that information to a target.

You can configure the following enrichments when setting up a pipe in EventBridge:

EventBridge invokes enrichments synchronously because it must wait for a response from the enrichment before invoking the target.

Enrichment responses are limited to a maximum size of 6MB.

You can also transform the data you receive from the source before sending it for enhancement. For more information, see Amazon EventBridge Pipes input transformation.

Filtering events using enrichment

EventBridge Pipes passes the enrichment responses directly to the configured target. This includes array responses for targets that support batches. For more information about batch behavior, see Amazon EventBridge Pipes batching and concurrency. You can also use your enrichment as a filter and pass fewer events than were received from the source. If you don’t want to invoke the target, return an empty response, such as "", {}, or [].

Targets

After the event data has been filtered and enriched, you can send it to a specific target, such as an Amazon Kinesis stream or an Amazon CloudWatch log group. For a list of the available targets, see Amazon EventBridge Pipes targets.

You can transform the data after it’s enhanced and before it’s sent to the target. For more information, see Amazon EventBridge Pipes input transformation.

Validating configuration parameters

After a pipe is created, EventBridge validates the following configuration parameters:

Invocation types

EventBridge has the following ways to invoke an enrichment or target:

By default, EventBridge invokes enrichments synchronously because it must wait for a response from the enrichment before invoking the target. Similarly, for pipes with ordered sources, targets are synchronously invoked because a response from the target is needed before proceeding to the next event.

If an source doesn't enforce order, such as a standard Amazon SQS queue, EventBridge can invoke a supported target synchronously or asynchronously.

With Lambda functions and Step Functions state machines, you can configure the invocation type.

Starting or stopping a pipe

By default, a pipe is Running and processes events when it's created. To create a pipe without processing events, set the DesiredState to Stopped in your API call or turn off the Activate pipe setting in the console.

If you create a pipe with Amazon SQS, Kinesis, or DynamoDB sources, pipe creation can typically take a minute or two.

If you create a pipe with Amazon MSK, self managed Apache Kafka, or Amazon MQ sources, pipes creation can take up to ten minutes.

To change the state of an existing pipe, do the following:

There can be a delay between when a pipe is STOPPED and when it no longer processes events. This delay is typically less than two minutes.