Using CloudFormation macros to perform custom processing on templates
Macros enable you to perform custom processing on templates, from simple actions like find-and-replace operations to extensive transformations of entire templates.
To get an idea of the breadth of possibilities, consider the AWS::Include
and
AWS::Serverless
transforms, which are macros hosted by CloudFormation:
-
AWS::Include transform enables you to insert boilerplate template snippets into your templates.
-
AWS::Serverless transform takes an entire template written in the Amazon Serverless Application Model (Amazon SAM) syntax and transforms and expands it into a compliant CloudFormation template. (For more information about serverless applications and Amazon SAM, see Deploying Lambda-based applications in the Amazon Lambda Developer Guide.)
How CloudFormation macros work
There are two major steps to processing templates using macros: creating the macro itself, and then using the macro to perform processing on your templates.
To create a macro definition, you need to create the following:
-
An Amazon Lambda function to perform the template processing. This Lambda function accepts either a snippet or an entire template, and any additional parameters that you define. It returns the processed template snippet or the entire template as a response.
-
A resource of type
AWS::CloudFormation::Macro
, which enables users to call the Lambda function from within CloudFormation templates. This resource specifies the ARN of the Lambda function to invoke for this macro, and additional optional properties to assist with debugging. To create this resource within an account, author a template that includes theAWS::CloudFormation::Macro
resource, and then create either a stack or stack set with self-managed permissions from the template. Amazon CloudFormation StackSets doesn't currently support creating or updating stack sets with service-managed permissions from templates that reference macros.
To use a macro, reference the macro in your template:
-
To process a section, or snippet, of a template, reference the macro in an
Fn::Transform
function located relative to the template content you want to transform. When usingFn::Transform
, you can also pass any specified parameters it requires. -
To process an entire template, reference the macro in the Transform section of the template.
Next, you typically create a change set and then execute it. (Processing macros can add multiple resources that you might not be aware of. To ensure that you're aware of all of the changes introduced by macros, we strongly advise that you use change sets.) CloudFormation passes the specified template content, along with any additional specified parameters, to the Lambda function specified in the macro resource. The Lambda function returns the processed template content, be it a snippet or an entire template.
After all macros in the template have been called, CloudFormation generates a change set that includes the processed template content. After you review the change set, execute it to apply the changes.
Important
If your stack set template references one or more macros, you must create the stack
set directly from the processed template, without first reviewing the resulting changes
in a change set. To create or update the stack set directly, you must use the CreateStackSet or UpdateStackSet action and specify the CAPABILITY_AUTO_EXPAND
capability. Before you create or update a stack set from a template that references
macros directly, be sure that you know what processing the macros performs.
![Use the Fn::Transform intrinsic function or the Transform section of the template, to pass the template contents and associated parameters to the macro's underlying Lambda function, which returns the processed template contents.](images/template-macro-use.png)
Note
If you are comfortable creating or updating a stack directly from a processed
template, without first reviewing the proposed changes in a change set, you can do so by
specifying the CAPABILITY_AUTO_EXPAND
capability during a
CreateStack
or UpdateStack
request. You should only create
stacks directly from a template that references macros if you know what processing the
macros performs.
For more information, see CreateStack or UpdateStack in the Amazon CloudFormation API Reference.
Creating a CloudFormation macro definition
When you create a macro definition, the macro definition makes the underlying Lambda function available in the specified account so that CloudFormation invokes it to process the templates.
CloudFormation macro function interface
For macros, CloudFormation invokes the underlying Lambda functions with the following event mapping. CloudFormation sends its request in JSON format, and it expects the function response to be formatted as JSON too.
{ "region" : "
us-east-1
", "accountId" : "$ACCOUNT_ID
", "fragment" : {...
}, "transformId" : "$TRANSFORM_ID
", "params" : {...
}, "requestId" : "$REQUEST_ID
", "templateParameterValues" : {...
} }
-
region
The Region in which the macro resides.
-
accountId
The account ID of the account from which the macro is invoking the Lambda function.
-
fragment
The template content available for custom processing, in JSON format.
-
For macros included in the
Transform
template section, this is the entire template except for theTransform
section. -
For macros included in an
Fn::Transform
intrinsic function call, this includes all sibling nodes (and their children) based on the location of the intrinsic function within the template except for theFn::Transform
function. For more information, see CloudFormation macro scope.
-
-
transformId
The name of the macro invoking this function.
-
params
For
Fn::Transform
function calls, any specified parameters for the function. CloudFormation doesn't evaluate these parameters before passing them to the function.For macros included in the
Transform
template section, this section is empty. -
requestId
The ID of the request invoking this function.
-
templateParameterValues
Any parameters specified in the Parameters section of the template. CloudFormation evaluates these parameters before passing them to the function.
CloudFormation expects the underlying function to return a response in the following JSON format:
{ "requestId" : "
$REQUEST_ID
", "status" : "$STATUS
", "fragment" : {...
} "errorMessage": "optional error message for failures" }
-
requestId
The ID of the request invoking this function. This must match the request ID provided by CloudFormation when invoking the function.
-
status
The status of the request (case-insensitive). Should be set to
success
. CloudFormation treats any other response as a failure. -
fragment
The processed template content for CloudFormation to include in the processed template, including siblings. CloudFormation replaces the template content that is passed to the Lambda function with the template fragment it receives in the Lambda response.
The processed template content must be valid JSON, and its inclusion in the processed template must result in a valid template.
If your function doesn't actually change the template content that CloudFormation passes to it, but you still need to include that content in the processed template, your function needs to return that template content to CloudFormation in its response.
-
errorMessage
The error message that explains why the transform failed. CloudFormation displays this error message in the Events pane of the Stack details page for your stack.
For example, "Error creating change set: Transform
Amazon Web Services account account number
::macro name
failed with:error message string
".
For information about additional considerations when creating macros, see Considerations when creating CloudFormation macro definitions.
CloudFormation macro account scope and permissions
You can use macros only in the account in which they were created as a resource. The name of the macro must be unique within a given account. However, you can make the same functionality available in multiple accounts by enabling cross-account access on the underlying Lambda function, and then creating macro definitions referencing that function in multiple accounts. In the example below, three accounts contain macro definitions that each point to the same Lambda function.
![By allowing cross-account access on the Lambda function, Amazon enables you to create macros in multiple accounts that reference that function.](images/template-macro-accounts.png)
For more information, see Overview of managing access permissions to your Amazon Lambda resources in the Amazon Lambda Developer Guide.
To create a macro definition, the user must have permissions to create a stack or stack set within the specified account.
For CloudFormation to successfully run a macro included in a template, the user must
have Invoke
permissions for the underlying Lambda function. To prevent
potential escalation of permissions, CloudFormation impersonates the user while running
the macro. For more information, see Lambda permissions model
in the Amazon Lambda Developer Guide and Actions and condition context keys for
Amazon Lambda in the IAM User Guide.
The AWS::Serverless transform and AWS::Include transform transforms are macros hosted by CloudFormation. No special permissions are necessary to use them, and they're available from within any account in CloudFormation.
Debugging CloudFormation macros
To aid in debugging, you can also specify the LogGroupName
and
LogRoleArn
properties when creating the
AWS::CloudFormation::Macro
resource type for your macro. These
properties enable you to specify the CloudWatch log group to which CloudFormation sends error
logging information when invoking the macro's underlying Amazon Lambda function, and the
role CloudFormation should assume when sending log entries to those logs.
Billing
When a macro runs, the owner of the Lambda function is billed for any charges related to the execution of that function.
The AWS::Serverless transform and AWS::Include transform transforms are macros hosted by CloudFormation. There is no charge for using them.
Considerations when creating CloudFormation macro definitions
When creating macro definitions, keep the following in mind:
-
Macros are supported only in Amazon Web Services Regions where Amazon Lambda is available. For a list of Regions where Lambda is available, see Amazon Lambda endpoints and quotas.
-
Any processed template snippets must be valid JSON.
-
Any processed template snippets must pass validation checks for a create stack, update stack, create stack set, or update stack set operation.
-
CloudFormation resolves macros first, and then processes the template. The resulting template must be valid JSON and must not exceed the template size limit.
-
Because of the order in which CloudFormation processes elements in a template, a macro can't include modules in the processed template content it returns to CloudFormation. For more information about modules, see Developing modules in the CloudFormation CLI User Guide.
-
When using the update rollback feature, CloudFormation uses a copy of the original template. It rolls back to the original template even if the included snippet was changed.
-
Including macros within macros doesn't work because we don't process macros recursively.
-
The
Fn::ImportValue
intrinsic function isn't currently supported in macros. -
Intrinsic functions included in the template are evaluated after any macros. Therefore, the processed template content your macro returns can include calls to intrinsic functions, and they're evaluated as usual.
-
StackSets doesn't currently support creating or updating stack sets with service-managed permissions from templates that reference CloudFormation macros.
-
If your stack set template references one or more macros, you must create or update the stack set directly from the processed template, without first reviewing the resulting changes in a change set. To create or update the stack set directly, use the CreateStackSet or UpdateStackSet action and specify the
CAPABILITY_AUTO_EXPAND
capability. Processing macros can add multiple resources that you might not be aware of. Before you create or update a stack set from a template that references macros directly, be sure that you know what processing the macros performs. -
Change sets don't currently support nested stacks. If you want to create or update a stack using a template that references macros and contains nested stacks, you must create or update the stack directly. To perform this, use the CreateStack or UpdateStack action and specify the
CAPABILITY_AUTO_EXPAND
capability.
To create a CloudFormation macro definition:
-
Build an Amazon Lambda function that processes CloudFormation templates.
The Lambda function you build performs the processing of the template contents. Your function can process any part of a template, up to the entire template. For information about the event mapping to which your function must adhere, see CloudFormation macro function interface. For information about additional considerations when creating macros, see Considerations when creating CloudFormation macro definitions.
-
Create a template containing an
AWS::CloudFormation::Macro
resource type.-
You must specify the
Name
andFunctionName
properties. TheFunctionName
property specifies the ARN of the Lambda function to invoke when CloudFormation runs the macro. -
To aid in debugging, you can also specify the
LogGroupName
andLogRoleArn
properties.
-
-
Create a stack from the template containing the macro in the desired account, or create a stack set with self-managed permissions from the template referencing the macro in the administrator account, then create stack instances in the desired target accounts.
After CloudFormation has successfully created stacks that contains the macro definition, the macro is available for use within those accounts.
Using CloudFormation macros in your templates
After CloudFormation has successfully created the stacks that contain the macro definition, the macro is available for use within those accounts. You use a macro by referencing it in a template, at the appropriate location relevant to the template contents you want to process.
CloudFormation macro evaluation order
You can reference multiple macros in a given template, including transforms hosted by CloudFormation, such as AWS::Include transform and AWS::Serverless transform.
Macros are evaluated in order, based on their location in the template, from the most deeply nested outward to the most general. Macros at the same location in the template are evaluated serially based on the order in which they're listed.
Transforms such as AWS::Include
and AWS::Transform
are
treated the same as any other macros in terms of action order and scope.
For example, in the template sample below, CloudFormation evaluates the
PolicyAdder
macro first, because it's the most deeply-nested macro
in the template. CloudFormation then evaluates MyMacro
before evaluating
AWS::Serverless
because it's listed before
AWS::Serverless
in the Transform
section.
AWSTemplateFormatVersion: 2010-09-09 Transform: [MyMacro, AWS::Serverless] Resources: WaitCondition: Type: AWS::CloudFormation::WaitCondition MyBucket: Type: 'AWS::S3::Bucket' Properties: BucketName:
amzn-s3-demo-bucket
Tags: [{"key":"value"}] 'Fn::Transform': - Name: PolicyAdder CorsConfiguration:[] MyEc2Instance: Type: 'AWS::EC2::Instance' Properties: ImageID: "ami-123"
CloudFormation macro scope
Macros referenced in the Transform
section of a template can process
the entire contents of that template.
Macros referenced in an Fn::Transform
function can process the
contents of any of the sibling elements (including children) of that
Fn::Transform
function in the template.
For example, in the template sample below, AWS::Include
can process
the MyBucket
properties, based on the location of the
Fn::Transform
function that contains it. MyMacro
can
process the contents of the entire template because of its inclusion in the
Transform
section.
// Start of processable content for MyMacro AWSTemplateFormatVersion: 2010-09-09 Transform: [MyMacro] Resources: WaitCondition: Type: AWS::CloudFormation::WaitCondition MyBucket: Type: 'AWS::S3::Bucket' //Start of processable content for AWS::Include Properties: BucketName:
amzn-s3-demo-bucket1
Tags: [{"key":"value"}] 'Fn::Transform': - Name: 'AWS::Include' Parameters: Location: s3://amzn-s3-demo-bucket2
/MyFileName.yaml CorsConfiguration:[] //End of processable content for AWS::Include MyEc2Instance: Type: 'AWS::EC2::Instance' Properties: ImageID: "ami-123" // End of processable content for MyMacro
Change sets and CloudFormation macros
To create or update a stack using a template that references macros, you typically create a change set and then execute it. A change set describes the actions CloudFormation will take based on the processed template. Processing macros can add multiple resources that you might not be aware of. To ensure that you're aware of all the changes introduced by macros, we strongly suggest you use change sets. After you review the change set, you can execute it to actually apply the changes.
A macro can add IAM resources to your template. For these resources, CloudFormation requires you to acknowledge their capabilities. Because CloudFormation can't know which resources are added before processing your template, you might need to acknowledge IAM capabilities when you create the change set, depending on whether the referenced macros contain IAM resources. That way, when you run the change set, CloudFormation has the necessary capabilities to create IAM resources.
Important
If your stack set template references one or more macros, you must create the
stack set directly from the processed template, without first reviewing the
resulting changes in a change set. To create or update the stack set directly,
you must use the CreateStackSet or UpdateStackSet action and specify the
CAPABILITY_AUTO_EXPAND
capability. Before you create or update
a stack set from a template that references macros directly, be sure that you
know what processing the macros performs.
Note
If you are comfortable creating or updating a stack directly from a processed
template, without first reviewing the proposed changes in a change set, you can
do so by specifying the CAPABILITY_AUTO_EXPAND
capability during a
CreateStack
or UpdateStack
request. You should
only create stacks directly from a stack template that contains macros if you
know what processing the macro performs. You can't use change sets with stack
set macros; you must update your stack set directly.
For more information, see CreateStack or UpdateStack in the Amazon CloudFormation API Reference.
If you use the Amazon CLI, you can use the package
and deploy
commands to reduce the number of steps for launching stacks from templates that
reference macros. For more information, see Deploying Lambda-based
applications in the Amazon Lambda Developer Guide.
Template stage and CloudFormation macros
A template's stage indicates whether the template is the original user-submitted template or one in which CloudFormation has processed the macros.
-
Original
: The template that the user originally submitted to create or update the stack or stack set. -
Processed
: The template CloudFormation used to create or update the stack or stack set after processing any referenced macros. The processed template is formatted as JSON, even if the original template was formatted as YAML.
Use the processed template for troubleshooting stack issues. If a template doesn't reference macros, the original and processed templates are identical.
You can use the CloudFormation console or Amazon CLI to see the stage of a stack template.
Note
The maximum size for a processed stack template is 51,200 bytes when passed
directly into a CreateStack
, UpdateStack
, or
ValidateTemplate
request, or 1 MB when passed as an S3 object
using an Amazon S3 template URL. However, during processing CloudFormation updates the
temporary state of the template as it serially processes the macros contained in
the template. Because of this, the size of the template during
processing may temporarily exceed the allowed size of a
fully-processed template. CloudFormation allows some buffer for these in-process
templates. However, you should design your templates and macros keeping in mind
the maximum allowed size for a processed stack template.
If CloudFormation returns a Transformation data limit exceeded
error
while processing your template, your template has exceeded the maximum template
size CloudFormation allows during processing.
To resolve this issue, consider doing the following:
-
Restructure your template into multiple templates to avoid exceeding the maximum size for in-process templates. For example:
-
Use nested stack templates to encapsulate parts of the template. For more information, see Working with nested stacks.
-
Create multiple stacks and use cross-stack references to exchange information between them. For more information, see Refer to resource outputs in another CloudFormation stack.
-
-
Reduce the size of template fragment returned by a given macro. CloudFormation doesn't tamper with the contents of fragments returned by macros.
To use a CloudFormation macro in your template
Note
For CloudFormation to successfully run a macro referenced in a template, the user
must have Invoke
permissions for the underlying Lambda function. For
more information, see Overview of managing access permissions to your Amazon Lambda
resources in the Amazon Lambda Developer Guide.
-
Include a reference to the macro in the template.
-
To process a template snippet, reference the macro in a
Fn::Transform
function located relative to the template content you want to process. -
To process the entire template, reference the macro in the Transform section of the template.
-
-
Create a change set using the template.
Important
If your stack set template references one or more macros, you must create the stack set directly from the processed template, without first reviewing the resulting changes in a change set. To create or update the stack set directly, you must use the CreateStackSet or UpdateStackSet action and specify the
CAPABILITY_AUTO_EXPAND
capability. Before you create or update a stack set from a template that references macros directly, be sure that you know what processing the macros performs. -
Review and run the change set.
Important
If your stack set template references one or more macros, you must create the stack set directly from the processed template, without first reviewing the resulting changes in a change set. To create or update the stack set directly, you must use the CreateStackSet or UpdateStackSet action and specify the
CAPABILITY_AUTO_EXPAND
capability. Before you create or update a stack set from a template that references macros directly, be sure that you know what processing the macros performs.
Macro examples
In addition to the Macro example: Creating and using a macro walkthrough in this guide,
you can find example macros, including source code and templates, in our GitHub repository