Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Fanout Amazon SNS events to Amazon Event Fork Pipelines

You can use Amazon SNS to build event-driven applications which use subscriber services to perform work automatically in response to events triggered by publisher services. This architectural pattern can make services more reusable, interoperable, and scalable. However, it can be labor-intensive to fork the processing of events into pipelines that address common event handling requirements, such as event storage, backup, search, analytics, and replay.

To accelerate the development of your event-driven applications, you can subscribe event-handling pipelines—powered by Amazon Event Fork Pipelines—to Amazon SNS topics. Amazon Event Fork Pipelines is a suite of open-source nested applications, based on the Amazon Serverless Application Model (Amazon SAM), which you can deploy directly from the Amazon Event Fork Pipelines suite (choose Show apps that create custom IAM roles or resource policies) into your Amazon account.

For an Amazon Event Fork Pipelines use case, see Deploying and testing the Amazon SNS event fork pipelines sample application.

How Amazon Event Fork Pipelines works

Amazon Event Fork Pipelines is a serverless design pattern. However, it is also a suite of nested serverless applications based on Amazon SAM (which you can deploy directly from the Amazon Serverless Application Repository (Amazon SAR) to your Amazon Web Services account in order to enrich your event-driven platforms). You can deploy these nested applications individually, as your architecture requires.

The following diagram shows an Amazon Event Fork Pipelines application supplemented by three nested applications. You can deploy any of the pipelines from the Amazon Event Fork Pipelines suite on the Amazon SAR independently, as your architecture requires.

Each pipeline is subscribed to the same Amazon SNS topic, allowing itself to process events in parallel as these events are published to the topic. Each pipeline is independent and can set its own Subscription Filter Policy. This allows a pipeline to process only a subset of the events that it is interested in (rather than all events published to the topic).

The event storage and backup pipeline

The following diagram shows the Event Storage and Backup Pipeline. You can subscribe this pipeline to your Amazon SNS topic to automatically back up the events flowing through your system.

This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, an Amazon Lambda function that automatically polls for these events in the queue and pushes them into an Amazon Data Firehose stream, and an Amazon S3 bucket that durably backs up the events loaded by the stream.

To fine-tune the behavior of your Firehose stream, you can configure it to buffer, transform, and compress your events prior to loading them into the bucket. As events are loaded, you can use Amazon Athena to query the bucket using standard SQL queries. You can also configure the pipeline to reuse an existing Amazon S3 bucket or create a new one.

The event search and analytics pipeline

The following diagram shows the Event Search and Analytics Pipeline. You can subscribe this pipeline to your Amazon SNS topic to index the events that flow through your system in a search domain and then run analytics on them.

This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, an Amazon Lambda function that polls events from the queue and pushes them into an Amazon Data Firehose stream, an Amazon OpenSearch Service domain that indexes the events loaded by the Firehose stream, and an Amazon S3 bucket that stores the dead-letter events that can’t be indexed in the search domain.

To fine-tune your Firehose stream in terms of event buffering, transformation, and compression, you can configure this pipeline.

You can also configure whether the pipeline should reuse an existing OpenSearch domain in your Amazon Web Services account or create a new one for you. As events are indexed in the search domain, you can use Kibana to run analytics on your events and update visual dashboards in real-time.

The event replay pipeline

The following diagram shows the Event Replay Pipeline. To record the events that have been processed by your system for the past 14 days (for example when your platform needs to recover from failure), you can subscribe this pipeline to your Amazon SNS topic and then reprocess the events.

This pipeline is comprised of an Amazon SQS queue that buffers the events delivered by the Amazon SNS topic, and an Amazon Lambda function that polls events from the queue and redrives them into your regular event processing pipeline, which is also subscribed to your topic.

Deploying Amazon Event Fork Pipelines

The Amazon Event Fork Pipelines suite (choose Show apps that create custom IAM roles or resource policies) is available as a group of public applications in the Amazon Serverless Application Repository, from where you can deploy and test them manually using the Amazon Lambda console. For information about deploying pipelines using the Amazon Lambda console, see Subscribing Amazon Event Fork Pipelines to an Amazon SNS topic.

In a production scenario, we recommend embedding Amazon Event Fork Pipelines within your overall application's Amazon SAM template. The nested-application feature lets you do this by adding the resource AWS::Serverless::Application to your Amazon SAM template, referencing the Amazon SAR ApplicationId and the SemanticVersion of the nested application.

For example, you can use the Event Storage and Backup Pipeline as a nested application by adding the following YAML snippet to the Resources section of your Amazon SAM template.

Backup:   
    Type: AWS::Serverless::Application
  Properties:
    Location:
      ApplicationId: arn:aws-cn:serverlessrepo:us-east-2:123456789012:applications/fork-event-storage-backup-pipeline
      SemanticVersion: 1.0.0
    Parameters: 
      #The ARN of the Amazon SNS topic whose messages should be backed up to the Amazon S3 bucket.
      TopicArn: !Ref MySNSTopic

When you specify parameter values, you can use Amazon CloudFormation intrinsic functions to reference other resources in your template. For example, in the YAML snippet above, the TopicArn parameter references the AWS::SNS::Topic resource MySNSTopic, defined elsewhere in the Amazon SAM template. For more information, see the Intrinsic Function Reference in the Amazon CloudFormation User Guide.