Create pipelines with an S3 source enabled for events (Amazon CloudFormation template) - Amazon CodePipeline
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Create pipelines with an S3 source enabled for events (Amazon CloudFormation template)

This procedure is for a pipeline where the source bucket has events enabled.

Use these steps to create a pipeline with an Amazon S3 source for event-based change detection.

To build an event-driven pipeline with Amazon S3, you edit the PollForSourceChanges parameter of your pipeline and then add the following resources to your template:

  • EventBridge rule and IAM role to allow this event to start your pipeline.

If you use Amazon CloudFormation to create and manage your pipelines, your template includes content like the following.

Note

The Configuration property in the source stage called PollForSourceChanges. If your template doesn't include that property, then PollForSourceChanges is set to true by default.

YAML
AppPipeline: Type: AWS::CodePipeline::Pipeline Properties: RoleArn: !GetAtt CodePipelineServiceRole.Arn Stages: - Name: Source Actions: - Name: SourceAction ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: S3 OutputArtifacts: - Name: SourceOutput Configuration: S3Bucket: !Ref SourceBucket S3ObjectKey: !Ref S3SourceObjectKey PollForSourceChanges: true RunOrder: 1 ...
JSON
"AppPipeline": { "Type": "AWS::CodePipeline::Pipeline", "Properties": { "RoleArn": { "Fn::GetAtt": ["CodePipelineServiceRole", "Arn"] }, "Stages": [ { "Name": "Source", "Actions": [ { "Name": "SourceAction", "ActionTypeId": { "Category": "Source", "Owner": "AWS", "Version": 1, "Provider": "S3" }, "OutputArtifacts": [ { "Name": "SourceOutput" } ], "Configuration": { "S3Bucket": { "Ref": "SourceBucket" }, "S3ObjectKey": { "Ref": "SourceObjectKey" }, "PollForSourceChanges": true }, "RunOrder": 1 } ] }, ...
To create an EventBridge rule with Amazon S3 as the event source and CodePipeline as the target and apply the permissions policy
  1. In the template, under Resources, use the AWS::IAM::Role Amazon CloudFormation resource to configure the IAM role that allows your event to start your pipeline. This entry creates a role that uses two policies:

    • The first policy allows the role to be assumed.

    • The second policy provides permissions to start the pipeline.

    Why am I making this change? Adding AWS::IAM::Role resource enables Amazon CloudFormation to create permissions for EventBridge. This resource is added to your Amazon CloudFormation stack.

    YAML
    EventRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: eb-pipeline-execution PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: codepipeline:StartPipelineExecution Resource: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref AppPipeline ] ] ...
    JSON
    "EventRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "events.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "Policies": [ { "PolicyName": "eb-pipeline-execution", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "codepipeline:StartPipelineExecution", "Resource": { "Fn::Join": [ "", [ "arn:aws:codepipeline:", { "Ref": "AWS::Region" }, ":", { "Ref": "AWS::AccountId" }, ":", { "Ref": "AppPipeline" } ] ] ...
  2. Use the AWS::Events::Rule Amazon CloudFormation resource to add an EventBridge rule. This event pattern creates an event that monitors creation or deletion of objects in your Amazon S3 source bucket. In addition, include a target of your pipeline. When an object is created, this rule invokes StartPipelineExecution on your target pipeline.

    Why am I making this change? Adding the AWS::Events::Rule resource enables Amazon CloudFormation to create the event. This resource is added to your Amazon CloudFormation stack.

    YAML
    EventRule: Type: AWS::Events::Rule Properties: EventBusName: default EventPattern: source: - aws.s3 detail-type: - Object Created detail: bucket: name: - !Ref SourceBucket Name: EnabledS3SourceRule State: ENABLED Targets: - Arn: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref AppPipeline ] ] RoleArn: !GetAtt EventRole.Arn Id: codepipeline-AppPipeline ...
    JSON
    "EventRule": { "Type": "AWS::Events::Rule", "Properties": { "EventBusName": "default", "EventPattern": { "source": [ "aws.s3" ], "detail-type": [ "Object Created" ], "detail": { "bucket": { "name": [ "s3-pipeline-source-fra-bucket" ] } } }, "Name": "EnabledS3SourceRule", "State": "ENABLED", "Targets": [ { "Arn": { "Fn::Join": [ "", [ "arn:aws:codepipeline:", { "Ref": "AWS::Region" }, ":", { "Ref": "AWS::AccountId" }, ":", { "Ref": "AppPipeline" } ] ] }, "RoleArn": { "Fn::GetAtt": [ "EventRole", "Arn" ] }, "Id": "codepipeline-AppPipeline" } ] } } }, ...
  3. Save your updated template to your local computer, and open the Amazon CloudFormation console.

  4. Choose your stack, and then choose Create Change Set for Current Stack.

  5. Upload your updated template, and then view the changes listed in Amazon CloudFormation. These are the changes that will be made to the stack. You should see your new resources in the list.

  6. Choose Execute.

To edit your pipeline's PollForSourceChanges parameter
Important

When you create a pipeline with this method, the PollForSourceChanges parameter defaults to true if it is not explicitly set to false. When you add event-based change detection, you must add the parameter to your output and set it to false to disable polling. Otherwise, your pipeline starts twice for a single source change. For details, see Valid settings for the PollForSourceChanges parameter.

  • In the template, change PollForSourceChanges to false. If you did not include PollForSourceChanges in your pipeline definition, add it and set it to false.

    Why am I making this change? Changing PollForSourceChanges to false turns off periodic checks so you can use event-based change detection only.

    YAML
    Name: Source Actions: - Name: SourceAction ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: S3 OutputArtifacts: - Name: SourceOutput Configuration: S3Bucket: !Ref SourceBucket S3ObjectKey: !Ref SourceObjectKey PollForSourceChanges: false RunOrder: 1
    JSON
    { "Name": "SourceAction", "ActionTypeId": { "Category": "Source", "Owner": "AWS", "Version": 1, "Provider": "S3" }, "OutputArtifacts": [ { "Name": "SourceOutput" } ], "Configuration": { "S3Bucket": { "Ref": "SourceBucket" }, "S3ObjectKey": { "Ref": "SourceObjectKey" }, "PollForSourceChanges": false }, "RunOrder": 1 }

When you use Amazon CloudFormation to create these resources, your pipeline is triggered when files in your repository are created or updated.

Note

Do not stop here. Although your pipeline is created, you must create a second Amazon CloudFormation template for your Amazon S3 pipeline. If you do not create the second template, your pipeline does not have any change detection functionality.

YAML
Parameters: SourceObjectKey: Description: 'S3 source artifact' Type: String Default: SampleApp_Linux.zip ApplicationName: Description: 'CodeDeploy application name' Type: String Default: DemoApplication BetaFleet: Description: 'Fleet configured in CodeDeploy' Type: String Default: DemoFleet Resources: SourceBucket: Type: AWS::S3::Bucket Properties: NotificationConfiguration: EventBridgeConfiguration: EventBridgeEnabled: true VersioningConfiguration: Status: Enabled CodePipelineArtifactStoreBucket: Type: AWS::S3::Bucket CodePipelineArtifactStoreBucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref CodePipelineArtifactStoreBucket PolicyDocument: Version: 2012-10-17 Statement: - Sid: DenyUnEncryptedObjectUploads Effect: Deny Principal: '*' Action: s3:PutObject Resource: !Join [ '', [ !GetAtt CodePipelineArtifactStoreBucket.Arn, '/*' ] ] Condition: StringNotEquals: s3:x-amz-server-side-encryption: aws:kms - Sid: DenyInsecureConnections Effect: Deny Principal: '*' Action: s3:* Resource: !Sub ${CodePipelineArtifactStoreBucket.Arn}/* Condition: Bool: aws:SecureTransport: false CodePipelineServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - codepipeline.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: AWS-CodePipeline-Service-3 PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - codecommit:CancelUploadArchive - codecommit:GetBranch - codecommit:GetCommit - codecommit:GetUploadArchiveStatus - codecommit:UploadArchive Resource: 'resource_ARN' - Effect: Allow Action: - codedeploy:CreateDeployment - codedeploy:GetApplicationRevision - codedeploy:GetDeployment - codedeploy:GetDeploymentConfig - codedeploy:RegisterApplicationRevision Resource: 'resource_ARN' - Effect: Allow Action: - codebuild:BatchGetBuilds - codebuild:StartBuild Resource: 'resource_ARN' - Effect: Allow Action: - devicefarm:ListProjects - devicefarm:ListDevicePools - devicefarm:GetRun - devicefarm:GetUpload - devicefarm:CreateUpload - devicefarm:ScheduleRun Resource: 'resource_ARN' - Effect: Allow Action: - lambda:InvokeFunction - lambda:ListFunctions Resource: 'resource_ARN' - Effect: Allow Action: - iam:PassRole Resource: 'resource_ARN' - Effect: Allow Action: - elasticbeanstalk:* - ec2:* - elasticloadbalancing:* - autoscaling:* - cloudwatch:* - s3:* - sns:* - cloudformation:* - rds:* - sqs:* - ecs:* Resource: 'resource_ARN' AppPipeline: Type: AWS::CodePipeline::Pipeline Properties: Name: s3-events-pipeline RoleArn: !GetAtt CodePipelineServiceRole.Arn Stages: - Name: Source Actions: - Name: SourceAction ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: S3 OutputArtifacts: - Name: SourceOutput Configuration: S3Bucket: !Ref SourceBucket S3ObjectKey: !Ref SourceObjectKey PollForSourceChanges: false RunOrder: 1 - Name: Beta Actions: - Name: BetaAction InputArtifacts: - Name: SourceOutput ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CodeDeploy Configuration: ApplicationName: !Ref ApplicationName DeploymentGroupName: !Ref BetaFleet RunOrder: 1 ArtifactStore: Type: S3 Location: !Ref CodePipelineArtifactStoreBucket EventRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: eb-pipeline-execution PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: codepipeline:StartPipelineExecution Resource: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref AppPipeline ] ] EventRule: Type: AWS::Events::Rule Properties: EventBusName: default EventPattern: source: - aws.s3 detail-type: - Object Created detail: bucket: name: - !Ref SourceBucket Name: EnabledS3SourceRule State: ENABLED Targets: - Arn: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref AppPipeline ] ] RoleArn: !GetAtt EventRole.Arn Id: codepipeline-AppPipeline
JSON
{ "Parameters": { "SourceObjectKey": { "Description": "S3 source artifact", "Type": "String", "Default": "SampleApp_Linux.zip" }, "ApplicationName": { "Description": "CodeDeploy application name", "Type": "String", "Default": "DemoApplication" }, "BetaFleet": { "Description": "Fleet configured in CodeDeploy", "Type": "String", "Default": "DemoFleet" } }, "Resources": { "SourceBucket": { "Type": "AWS::S3::Bucket", "Properties": { "NotificationConfiguration": { "EventBridgeConfiguration": { "EventBridgeEnabled": true } }, "VersioningConfiguration": { "Status": "Enabled" } } }, "CodePipelineArtifactStoreBucket": { "Type": "AWS::S3::Bucket" }, "CodePipelineArtifactStoreBucketPolicy": { "Type": "AWS::S3::BucketPolicy", "Properties": { "Bucket": { "Ref": "CodePipelineArtifactStoreBucket" }, "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "DenyUnEncryptedObjectUploads", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "CodePipelineArtifactStoreBucket", "Arn" ] }, "/*" ] ] }, "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": "aws:kms" } } }, { "Sid": "DenyInsecureConnections", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "CodePipelineArtifactStoreBucket", "Arn" ] }, "/*" ] ] }, "Condition": { "Bool": { "aws:SecureTransport": false } } } ] } } }, "CodePipelineServiceRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "codepipeline.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "Policies": [ { "PolicyName": "AWS-CodePipeline-Service-3", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codecommit:CancelUploadArchive", "codecommit:GetBranch", "codecommit:GetCommit", "codecommit:GetUploadArchiveStatus", "codecommit:UploadArchive" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "codedeploy:CreateDeployment", "codedeploy:GetApplicationRevision", "codedeploy:GetDeployment", "codedeploy:GetDeploymentConfig", "codedeploy:RegisterApplicationRevision" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "codebuild:BatchGetBuilds", "codebuild:StartBuild" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "devicefarm:ListProjects", "devicefarm:ListDevicePools", "devicefarm:GetRun", "devicefarm:GetUpload", "devicefarm:CreateUpload", "devicefarm:ScheduleRun" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:ListFunctions" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": "resource_ARN" }, { "Effect": "Allow", "Action": [ "elasticbeanstalk:*", "ec2:*", "elasticloadbalancing:*", "autoscaling:*", "cloudwatch:*", "s3:*", "sns:*", "cloudformation:*", "rds:*", "sqs:*", "ecs:*" ], "Resource": "resource_ARN" } ] } } ] } }, "AppPipeline": { "Type": "AWS::CodePipeline::Pipeline", "Properties": { "Name": "s3-events-pipeline", "RoleArn": { "Fn::GetAtt": [ "CodePipelineServiceRole", "Arn" ] }, "Stages": [ { "Name": "Source", "Actions": [ { "Name": "SourceAction", "ActionTypeId": { "Category": "Source", "Owner": "AWS", "Version": 1, "Provider": "S3" }, "OutputArtifacts": [ { "Name": "SourceOutput" } ], "Configuration": { "S3Bucket": { "Ref": "SourceBucket" }, "S3ObjectKey": { "Ref": "SourceObjectKey" }, "PollForSourceChanges": false }, "RunOrder": 1 } ] }, { "Name": "Beta", "Actions": [ { "Name": "BetaAction", "InputArtifacts": [ { "Name": "SourceOutput" } ], "ActionTypeId": { "Category": "Deploy", "Owner": "AWS", "Version": 1, "Provider": "CodeDeploy" }, "Configuration": { "ApplicationName": { "Ref": "ApplicationName" }, "DeploymentGroupName": { "Ref": "BetaFleet" } }, "RunOrder": 1 } ] } ], "ArtifactStore": { "Type": "S3", "Location": { "Ref": "CodePipelineArtifactStoreBucket" } } } }, "EventRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "events.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "Policies": [ { "PolicyName": "eb-pipeline-execution", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "codepipeline:StartPipelineExecution", "Resource": { "Fn::Join": [ "", [ "arn:aws:codepipeline:", { "Ref": "AWS::Region" }, ":", { "Ref": "AWS::AccountId" }, ":", { "Ref": "AppPipeline" } ] ] } } ] } } ] } }, "EventRule": { "Type": "AWS::Events::Rule", "Properties": { "EventBusName": "default", "EventPattern": { "source": [ "aws.s3" ], "detail-type": [ "Object Created" ], "detail": { "bucket": { "name": [ { "Ref": "SourceBucket" } ] } } }, "Name": "EnabledS3SourceRule", "State": "ENABLED", "Targets": [ { "Arn": { "Fn::Join": [ "", [ "arn:aws:codepipeline:", { "Ref": "AWS::Region" }, ":", { "Ref": "AWS::AccountId" }, ":", { "Ref": "AppPipeline" } ] ] }, "RoleArn": { "Fn::GetAtt": [ "EventRole", "Arn" ] }, "Id": "codepipeline-AppPipeline" } ] } } } } }