Namespace Amazon.CDK.AWS.S3.Deployment
AWS S3 Deployment Construct Library
This library allows populating an S3 bucket with the contents of .zip files from other S3 buckets or from local disk.
The following example defines a publicly accessible S3 bucket with web hosting enabled and populates it from a local directory on disk.
var websiteBucket = new Bucket(this, "WebsiteBucket", new BucketProps {
WebsiteIndexDocument = "index.html",
PublicReadAccess = true
});
new BucketDeployment(this, "DeployWebsite", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website-dist") },
DestinationBucket = websiteBucket,
DestinationKeyPrefix = "web/static"
});
This is what happens under the hood:
If you are referencing the filled bucket in another construct that depends on
the files already be there, be sure to use deployment.deployedBucket
. This
will ensure the bucket deployment has finished before the resource that uses
the bucket is created:
Bucket websiteBucket;
var deployment = new BucketDeployment(this, "DeployWebsite", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "my-website")) },
DestinationBucket = websiteBucket
});
new ConstructThatReadsFromTheBucket(this, "Consumer", new Dictionary<string, IBucket> {
// Use 'deployment.deployedBucket' instead of 'websiteBucket' here
{ "bucket", deployment.DeployedBucket }
});
It is also possible to add additional sources using the addSource
method.
IBucket websiteBucket;
var deployment = new BucketDeployment(this, "DeployWebsite", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website-dist") },
DestinationBucket = websiteBucket,
DestinationKeyPrefix = "web/static"
});
deployment.AddSource(Source.Asset("./another-asset"));
For the Lambda function to download object(s) from the source bucket, besides the obvious
s3:GetObject*
permissions, the Lambda's execution role needs the kms:Decrypt
and kms:DescribeKey
permissions on the KMS key that is used to encrypt the bucket. By default, when the source bucket is
encrypted with the S3 managed key of the account, these permissions are granted by the key's
resource-based policy, so they do not need to be on the Lambda's execution role policy explicitly.
However, if the encryption key is not the s3 managed one, its resource-based policy is quite likely
to NOT grant such KMS permissions. In this situation, the Lambda execution will fail with an error
message like below:
download failed: ...
An error occurred (AccessDenied) when calling the GetObject operation:
User: *** is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext
because no identity-based policy allows the kms:Decrypt action
When this happens, users can use the public handlerRole
property of BucketDeployment
to manually
add the KMS permissions:
Bucket destinationBucket;
var deployment = new BucketDeployment(this, "DeployFiles", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "source-files")) },
DestinationBucket = destinationBucket
});
deployment.HandlerRole.AddToPolicy(
new PolicyStatement(new PolicyStatementProps {
Actions = new [] { "kms:Decrypt", "kms:DescribeKey" },
Effect = Effect.ALLOW,
Resources = new [] { "<encryption key ARN>" }
}));
The situation above could arise from the following scenarios:
Supported sources
The following source types are supported for bucket deployments:
To create a source from a single file, you can pass AssetOptions
to exclude
all but a single file:
IMPORTANT The aws-s3-deployment
module is only intended to be used with
zip files from trusted sources. Directories bundled by the CDK CLI (by using
Source.asset()
on a directory) are safe. If you are using Source.asset()
or
Source.bucket()
to reference an existing zip file, make sure you trust the
file you are referencing. Zips from untrusted sources might be able to execute
arbitrary code in the Lambda Function used by this module, and use its permissions
to read or write unexpected files in the S3 bucket.
Retain on Delete
By default, the contents of the destination bucket will not be deleted when the
BucketDeployment
resource is removed from the stack or when the destination is
changed. You can use the option retainOnDelete: false
to disable this behavior,
in which case the contents will be deleted.
Configuring this has a few implications you should be aware of:
General Recommendations
Shared Bucket
If the destination bucket is not dedicated to the specific BucketDeployment
construct (i.e shared by other entities),
we recommend to always configure the destinationKeyPrefix
property. This will prevent the deployment from
accidentally deleting data that wasn't uploaded by it.
Dedicated Bucket
If the destination bucket is dedicated, it might be reasonable to skip the prefix configuration,
in which case, we recommend to remove retainOnDelete: false
, and instead, configure the
autoDeleteObjects
property on the destination bucket. This will avoid the logical ID problem mentioned above.
Prune
By default, files in the destination bucket that don't exist in the source will be deleted
when the BucketDeployment
resource is created or updated. You can use the option prune: false
to disable
this behavior, in which case the files will not be deleted.
Bucket destinationBucket;
new BucketDeployment(this, "DeployMeWithoutDeletingFilesOnDestination", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "my-website")) },
DestinationBucket = destinationBucket,
Prune = false
});
This option also enables you to multiple bucket deployments for the same destination bucket & prefix, each with its own characteristics. For example, you can set different cache-control headers based on file extensions:
Bucket destinationBucket;
new BucketDeployment(this, "BucketDeployment", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website", new AssetOptions { Exclude = new [] { "index.html" } }) },
DestinationBucket = destinationBucket,
CacheControl = new [] { CacheControl.MaxAge(Duration.Days(365)), CacheControl.Immutable() },
Prune = false
});
new BucketDeployment(this, "HTMLBucketDeployment", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website", new AssetOptions { Exclude = new [] { "*", "!index.html" } }) },
DestinationBucket = destinationBucket,
CacheControl = new [] { CacheControl.MaxAge(Duration.Seconds(0)) },
Prune = false
});
Exclude and Include Filters
There are two points at which filters are evaluated in a deployment: asset bundling and the actual deployment. If you simply want to exclude files in the asset bundling process, you should leverage the exclude
property of AssetOptions
when defining your source:
Bucket destinationBucket;
new BucketDeployment(this, "HTMLBucketDeployment", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website", new AssetOptions { Exclude = new [] { "*", "!index.html" } }) },
DestinationBucket = destinationBucket
});
If you want to specify filters to be used in the deployment process, you can use the exclude
and include
filters on BucketDeployment
. If excluded, these files will not be deployed to the destination bucket. In addition, if the file already exists in the destination bucket, it will not be deleted if you are using the prune
option:
Bucket destinationBucket;
new BucketDeployment(this, "DeployButExcludeSpecificFiles", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "my-website")) },
DestinationBucket = destinationBucket,
Exclude = new [] { "*.txt" }
});
These filters follow the same format that is used for the AWS CLI. See the CLI documentation for information on Using Include and Exclude Filters.
Objects metadata
You can specify metadata to be set on all the objects in your deployment.
There are 2 types of metadata in S3: system-defined metadata and user-defined metadata.
System-defined metadata have a special purpose, for example cache-control defines how long to keep an object cached.
User-defined metadata are not used by S3 and keys always begin with x-amz-meta-
(this prefix is added automatically).
System defined metadata keys include the following:
You can find more information about system defined metadata keys in
S3 PutObject documentation
and aws s3 sync
documentation.
var websiteBucket = new Bucket(this, "WebsiteBucket", new BucketProps {
WebsiteIndexDocument = "index.html",
PublicReadAccess = true
});
new BucketDeployment(this, "DeployWebsite", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website-dist") },
DestinationBucket = websiteBucket,
DestinationKeyPrefix = "web/static", // optional prefix in destination bucket
Metadata = new Dictionary<string, string> { { "A", "1" }, { "b", "2" } }, // user-defined metadata
// system-defined metadata
ContentType = "text/html",
ContentLanguage = "en",
StorageClass = StorageClass.INTELLIGENT_TIERING,
ServerSideEncryption = ServerSideEncryption.AES_256,
CacheControl = new [] { CacheControl.SetPublic(), CacheControl.MaxAge(Duration.Hours(1)) },
AccessControl = BucketAccessControl.BUCKET_OWNER_FULL_CONTROL
});
CloudFront Invalidation
You can provide a CloudFront distribution and optional paths to invalidate after the bucket deployment finishes.
using Amazon.CDK.AWS.CloudFront;
using Amazon.CDK.AWS.CloudFront.Origins;
var bucket = new Bucket(this, "Destination");
// Handles buckets whether or not they are configured for website hosting.
var distribution = new Distribution(this, "Distribution", new DistributionProps {
DefaultBehavior = new BehaviorOptions { Origin = new S3Origin(bucket) }
});
new BucketDeployment(this, "DeployWithInvalidation", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website-dist") },
DestinationBucket = bucket,
Distribution = distribution,
DistributionPaths = new [] { "/images/*.png" }
});
Signed Content Payloads
By default, deployment uses streaming uploads which set the x-amz-content-sha256
request header to UNSIGNED-PAYLOAD
(matching default behavior of the AWS CLI tool).
In cases where bucket policy restrictions require signed content payloads, you can enable
generation of a signed x-amz-content-sha256
request header with signContent: true
.
IBucket bucket;
new BucketDeployment(this, "DeployWithSignedPayloads", new BucketDeploymentProps {
Sources = new [] { Source.Asset("./website-dist") },
DestinationBucket = bucket,
SignContent = true
});
Size Limits
The default memory limit for the deployment resource is 128MiB. If you need to
copy larger files, you can use the memoryLimit
configuration to increase the
size of the AWS Lambda resource handler.
The default ephemeral storage size for the deployment resource is 512MiB. If you
need to upload larger files, you may hit this limit. You can use the
ephemeralStorageSize
configuration to increase the storage size of the AWS Lambda
resource handler.
NOTE: a new AWS Lambda handler will be created in your stack for each combination
of memory and storage size.
EFS Support
If your workflow needs more disk space than default (512 MB) disk space, you may attach an EFS storage to underlying
lambda function. To Enable EFS support set efs
and vpc
props for BucketDeployment.
Check sample usage below. Please note that creating VPC inline may cause stack deletion failures. It is shown as below for simplicity. To avoid such condition, keep your network infra (VPC) in a separate stack and pass as props.
Bucket destinationBucket;
Vpc vpc;
new BucketDeployment(this, "DeployMeWithEfsStorage", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "my-website")) },
DestinationBucket = destinationBucket,
DestinationKeyPrefix = "efs/",
UseEfs = true,
Vpc = vpc,
RetainOnDelete = false
});
Data with deploy-time values
The content passed to Source.data()
, Source.jsonData()
, or Source.yamlData()
can include
references that will get resolved only during deployment. Only a subset of CloudFormation functions
are supported however, namely: Ref, Fn::GetAtt, Fn::Join, and Fn::Select (Fn::Split may be nested under Fn::Select).
For example:
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.ElasticLoadBalancingV2;
Bucket destinationBucket;
Topic topic;
ApplicationTargetGroup tg;
IDictionary<string, object> appConfig = new Dictionary<string, object> {
{ "topic_arn", topic.TopicArn },
{ "base_url", "https://my-endpoint" },
{ "lb_name", tg.FirstLoadBalancerFullName }
};
new BucketDeployment(this, "BucketDeployment", new BucketDeploymentProps {
Sources = new [] { Source.JsonData("config.json", appConfig) },
DestinationBucket = destinationBucket
});
The value in topic.topicArn
is a deploy-time value. It only gets resolved
during deployment by placing a marker in the generated source file and
substituting it when its deployed to the destination with the actual value.
Substitutions from Templated Files
The DeployTimeSubstitutedFile
construct allows you to specify substitutions
to make from placeholders in a local file which will be resolved during deployment. This
is especially useful in situations like creating an API from a spec file, where users might
want to reference other CDK resources they have created.
The syntax for template variables is {{ variableName }}
in your local file. Then, you would
specify the substitutions in CDK like this:
using Amazon.CDK.AWS.Lambda;
Function myLambdaFunction;
Bucket destinationBucket;
//(Optional) if provided, the resulting processed file would be uploaded to the destinationBucket under the destinationKey name.
string destinationKey;
Role role;
new DeployTimeSubstitutedFile(this, "MyFile", new DeployTimeSubstitutedFileProps {
Source = "my-file.yaml",
DestinationKey = destinationKey,
DestinationBucket = destinationBucket,
Substitutions = new Dictionary<string, string> {
{ "variableName", myLambdaFunction.FunctionName }
},
Role = role
});
Nested variables, like {{ {{ foo }} }}
or {{ foo {{ bar }} }}
, are not supported by this
construct. In the first case of a single variable being is double nested {{ {{ foo }} }}
, only
the {{ foo }}
would be replaced by the substitution, and the extra brackets would remain in the file.
In the second case of two nexted variables {{ foo {{ bar }} }}
, only the {{ bar }}
would be replaced
in the file.
Keep Files Zipped
By default, files are zipped, then extracted into the destination bucket.
You can use the option extract: false
to disable this behavior, in which case, files will remain in a zip file when deployed to S3. To reference the object keys, or filenames, which will be deployed to the bucket, you can use the objectKeys
getter on the bucket deployment.
using Amazon.CDK;
Bucket destinationBucket;
var myBucketDeployment = new BucketDeployment(this, "DeployMeWithoutExtractingFilesOnDestination", new BucketDeploymentProps {
Sources = new [] { Source.Asset(Join(__dirname, "my-website")) },
DestinationBucket = destinationBucket,
Extract = false
});
new CfnOutput(this, "ObjectKey", new CfnOutputProps {
Value = Fn.Select(0, myBucketDeployment.ObjectKeys)
});
Notes
Development
The custom resource is implemented in Python 3.9 in order to be able to leverage
the AWS CLI for "aws s3 sync". The code is under lib/lambda
and
unit tests are under test/lambda
.
This package requires Python 3.9 during build time in order to create the custom resource Lambda bundle and test it. It also relies on a few bash scripts, so might be tricky to build on Windows.
Roadmap
Classes
BucketDeployment |
|
BucketDeploymentProps | Properties for |
CacheControl | Used for HTTP cache-control header, which influences downstream caches. |
DeploymentSourceContext | Bind context for ISources. |
DeployTimeSubstitutedFile |
|
DeployTimeSubstitutedFileProps | |
ServerSideEncryption | Indicates whether server-side encryption is enabled for the object, and whether that encryption is from the AWS Key Management Service (AWS KMS) or from Amazon S3 managed encryption (SSE-S3). |
Source | Specifies bucket deployment source. |
SourceConfig | Source information. |
StorageClass | Storage class used for storing the object. |
UserDefinedObjectMetadata | (deprecated) Custom user defined metadata. |
Interfaces
IBucketDeploymentProps | Properties for |
IDeploymentSourceContext | Bind context for ISources. |
IDeployTimeSubstitutedFileProps | |
ISource | Represents a source for bucket deployments. |
ISourceConfig | Source information. |
IUserDefinedObjectMetadata | (deprecated) Custom user defined metadata. |