# Use the EC2 TOE component document framework for custom components To build a component using the EC2 Task Orchestrator and Executor (EC2 TOE) component framework, you must provide a YAML-based document that represents the phases and steps that apply for the component you create. Amazon Web Services services use your component when they create a new Amazon Machine Image (AMI) or container image. **Topics** + [ ## Component document workflow ](#component-doc-workflow) + [ ## Component logging ](#component-logging) + [ ## Input and output chaining ](#document-chaining) + [ ## Document schema and definitions ](#document-schema) + [ ## Document examples ](#document-example) + [ # Use variables in your custom component document ](toe-user-defined-variables.md) + [ # Use conditional constructs in EC2 TOE ](toe-conditional-constructs.md) + [ # Use comparison operators in EC2 TOE component documents ](toe-comparison-operators.md) + [ # Use logical operators in EC2 TOE component documents ](toe-logical-operators.md) + [ # Use looping constructs in EC2 TOE ](toe-looping-constructs.md) ## Component document workflow The EC2 TOE component document uses phases and steps to group related tasks, and organize those tasks into a logical workflow for the component. **Tip** The service that uses your component to build an image might implement rules about what phases to use for their build process, and when those phases are allowed to run. This is important to consider when you design your component. **Phases** Phases represent the progression of your workflow through the image build process. For example, the Image Builder service uses `build` and `validate` phases during its *build stage* for the images it produces. It uses the `test` and `container-host-test` phases during its *test stage* to ensure that the image snapshot or container image produces the expected results before creating the final AMI or distributing the container image. When the component runs, the associated commands for each phase are applied in the order that they appear in the component document. **Rules for phases** + Each phase name must be unique within a document. + You can define many phases in your document. + You must include at least one of the following phases in your document: + **build** – for Image Builder, this phase is generally used during the *build stage*. + **validate** – for Image Builder, this phase is generally used during the *build stage*. + **test** – for Image Builder, this phase is generally used during the *test stage*. + Phases always run in the order that they are defined in the document. The order in which they are specified for EC2 TOE commands in the Amazon CLI has no effect. **Steps** Steps are individual units of work that define the workflow within each phase. Steps run in sequential order. However, input or output for one step can also feed into a subsequent step as input. This is called "chaining". **Rules for steps** + The step name must be unique for the phase. + The step must use a supported action (action module) that returns an exit code. For a complete list of supported action modules, how they work, input/output values, and examples, see [Action modules supported by EC2 TOE component manager](toe-action-modules.md). ## Component logging EC2 TOE creates a new log folder on the EC2 instances that are used for building and testing a new image, each time your component runs. For container images, the log folder is stored in the container. To assist with troubleshooting if something goes wrong during the image creation process, the input document and all of the output files EC2 TOE creates while running the component are stored in the log folder. The log folder name is comprised of the following parts: 1. **Log directory** – when a service runs a EC2 TOE component, it passes in the log directory, along with other settings for the command. For the following examples, we show the log file format that Image Builder uses. + **Linux and macOS**: `/var/lib/amazon/toe/` + **Windows**: `$env:ProgramFiles\Amazon\TaskOrchestratorAndExecutor\` 1. **File prefix** – This is a standard prefix used for all components: "`TOE_`". 1. **Run time** – This is a timestamp in YYYY-MM-DD\$1HH-MM-SS\$1UTC-0 format. 1. **Execution ID** – This is the GUID that is assigned when EC2 TOE runs one or more components. Example: `/var/lib/amazon/toe/TOE_2021-07-01_12-34-56_UTC-0_a1bcd2e3-45f6-789a-bcde-0fa1b2c3def4` EC2 TOE stores the following core files in the log folder: **Input files** + **document.yaml** – The document that is used as input for the command. After the component runs, this file is stored as an artifact. **Output files** + **application.log** – The application log contains timestamped debug level information from EC2 TOE about what's happening as the component is running. + **detailedoutput.json** – This JSON file has detailed information about run status, inputs, outputs, and failures for all documents, phases, and steps that apply for the component as it runs. + **console.log** – The console log contains all of the standard out (stdout) and standard error (stderr) information that EC2 TOE writes to the console while the component is running. + **chaining.json** – This JSON file represents optimizations that EC2 TOE applied to resolve chaining expressions. **Note** The log folder might also contain other temporary files that are not covered here. ## Input and output chaining The EC2 TOE configuration management application provides a feature for chaining inputs and outputs by writing references in the following formats: `{{ phase_name.step_name.inputs/outputs.variable }}` or `{{ phase_name.step_name.inputs/outputs[index].variable }}` The chaining feature allows you to recycle code and improve the maintainability of the document. **Rules for chaining** + Chaining expressions can be used only in the inputs section of each step. + Statements with chaining expressions must be enclosed in quotes. For example: + **Invalid expression**: `echo {{ phase.step.inputs.variable }}` + **Valid expression**: `"echo {{ phase.step.inputs.variable }}"` + **Valid expression**: `'echo {{ phase.step.inputs.variable }}'` + Chaining expressions can reference variables from other steps and phases in the same document. However, the calling service might have rules that require chaining expressions to operate only within the context of a single stage. For example, Image Builder does not support chaining from the *build stage* to the *test stage*, as it runs each stage independently. + Indexes in chaining expressions follow zero-based indexing. The index starts with zero (0) to reference the first element. **Examples** To refer to the source variable in the second entry of the following example step, the chaining pattern is `{{ build.SampleS3Download.inputs[1].source }}`. ``` phases: - name: 'build' steps: - name: SampleS3Download action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://sample-bucket/sample1.ps1' destination: 'C:\sample1.ps1' - source: 's3://sample-bucket/sample2.ps1' destination: 'C:\sample2.ps1' ``` To refer to the output variable (equal to "Hello") of the following example step, the chaining pattern is `{{ build.SamplePowerShellStep.outputs.stdout }}`. ``` phases: - name: 'build' steps: - name: SamplePowerShellStep action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: commands: - 'Write-Host "Hello"' ``` ## Document schema and definitions The following is the YAML schema for a document. ``` name: (optional) description: (optional) schemaVersion: "string" phases: - name: "string" steps: - name: "string" action: "string" timeoutSeconds: integer onFailure: "Abort|Continue|Ignore" maxAttempts: integer inputs: ``` The schema definitions for a document are as follows. | Field | Description | Type | Required | | --- | --- | --- | --- | | name | Name of the document. | String | No | | description | Description of the document. | String | No | | schemaVersion | Schema version of the document, currently 1.0. | String | Yes | | phases | A list of phases with their steps. | List | Yes | The schema definitions for a phase are as follows. | Field | Description | Type | Required | | --- | --- | --- | --- | | name | Name of the phase. | String | Yes | | steps | List of the steps in the phase. | List | Yes | The schema definitions for a step are as follows. | Field | Description | Type | Required | Default value | | --- | --- | --- | --- | --- | | name | User-defined name for the step. | String | | | | action | Keyword pertaining to the module that runs the step. | String | | | | timeoutSeconds | Number of seconds that the step runs before failing or retrying. Also, supports -1 value, which indicates infinite timeout. 0 and other negative values are not allowed. | Integer | No | 7,200 sec (120 mins) | | onFailure | Specifies what the step should do in case of failure. Valid values are as follows: [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/imagebuilder/latest/userguide/toe-use-documents.html) | String | No | Abort | | maxAttempts | Maximum number of attempts allowed before failing the step. | Integer | No | 1 | | inputs | Contains parameters required by the action module to run the step. | Dict | Yes | | ## Document examples The following examples show AWSTOE component documents that perform tasks for the target operating system. ------ #### [ Linux ] **Example 1: Run a custom binary file** The following is an example document that downloads and runs a custom binary file on a Linux instance. ``` name: LinuxBin description: Download and run a custom Linux binary file. schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://amzn-s3-demo-source-bucket/myapplication destination: /tmp/myapplication - name: Enable action: ExecuteBash onFailure: Continue inputs: commands: - 'chmod u+x {{ build.Download.inputs[0].destination }}' - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '--install' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ #### [ Windows ] **Example 1: Install Windows updates** The following is an example document that installs all available Windows updates, runs a configuration script, validates the changes before the AMI is created, and tests the changes after the AMI is created. ``` name: RunConfig_UpdateWindows description: 'This document will install all available Windows updates and run a config script. It will then validate the changes before an AMI is created. Then after AMI creation, it will test all the changes.' schemaVersion: 1.0 phases: - name: build steps: - name: DownloadConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/config.ps1' destination: 'C:\config.ps1' - name: RunConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{build.DownloadConfigScript.inputs[0].destination}}' - name: Cleanup action: DeleteFile onFailure: Abort maxAttempts: 3 inputs: - path: '{{build.DownloadConfigScript.inputs[0].destination}}' - name: RebootAfterConfigApplied action: Reboot inputs: delaySeconds: 60 - name: InstallWindowsUpdates action: UpdateOS - name: validate steps: - name: DownloadTestConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/testConfig.ps1' destination: 'C:\testConfig.ps1' - name: ValidateConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{validate.DownloadTestConfigScript.inputs[0].destination}}' - name: Cleanup action: DeleteFile onFailure: Abort maxAttempts: 3 inputs: - path: '{{validate.DownloadTestConfigScript.inputs[0].destination}}' - name: test steps: - name: DownloadTestConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/testConfig.ps1' destination: 'C:\testConfig.ps1' - name: ValidateConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{test.DownloadTestConfigScript.inputs[0].destination}}' ``` **Example 2: Install the Amazon CLI on a Windows instance** The following is an example document that installs the Amazon CLI on a Windows instance, using the setup file. ``` name: InstallCLISetUp description: Install &CLI; using the setup file schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://aws-cli/AWSCLISetup.exe destination: C:\Windows\temp\AWSCLISetup.exe - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '/install' - '/quiet' - '/norestart' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` **Example 3: Install the Amazon CLI with the MSI installer** The following is an example document that installs the Amazon CLI with the MSI installer. ``` name: InstallCLIMSI description: Install &CLI; using the MSI installer schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://aws-cli/AWSCLI64PY3.msi destination: C:\Windows\temp\AWSCLI64PY3.msi - name: Install action: ExecuteBinary onFailure: Continue inputs: path: 'C:\Windows\System32\msiexec.exe' arguments: - '/i' - '{{ build.Download.inputs[0].destination }}' - '/quiet' - '/norestart' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ #### [ macOS ] **Example 1: Run a custom macOS binary file** The following is an example document that downloads and runs a custom binary file on a macOS instance. ``` name: macOSBin description: Download and run a binary file on macOS. schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://amzn-s3-demo-source-bucket/myapplication destination: /tmp/myapplication - name: Enable action: ExecuteBash onFailure: Continue inputs: commands: - 'chmod u+x {{ build.Download.inputs[0].destination }}' - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '--install' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ # Use variables in your custom component document Variables provide a way to label data with meaningful names that can be used throughout an application. You can define custom variables with simple and readable formats for complex workflows, and reference them in the YAML application component document for an EC2 TOE component. This section provides information to help you define variables for your EC2 TOE component in the YAML application component document, including syntax, name constraints, and examples. ## Constants Constants are immutable variables that cannot be modified or overridden once defined. Constants can be defined using values in the `constants` section of an EC2 TOE document. **Rules for constant names** + The name must be between 3 and 128 characters in length. + The name can only contain alphanumeric characters (a-z, A-Z, 0-9), dashes (-), or underscores (\$1). + The name must be unique within the document. + The name must be specified as a YAML string. **Syntax** ``` constants: - : type: value: ``` | Key name | Required | Description | | --- | --- | --- | | `name` | Yes | Name of the constant. Must be unique for the document (it must not be the same as any other parameter names or constants). | | `value` | Yes | Value of the constant. | | `type` | Yes | Type of the constant. Supported type is string. | **Reference constant values in a document** You can reference constants in step or loop inputs inside of your YAML document, as follows: + Constant references are case-sensitive, and the name must match exactly. + The name must be enclosed within double curly braces `{{` *MyConstant* `}}`. + Spaces are allowed within the curly braces, and are automatically trimmed. For example, all of the following references are valid: `{{ MyConstant }}`, `{{ MyConstant}}`, `{{MyConstant }}`, `{{MyConstant}}` + The reference in the YAML document must be specified as a string (enclosed in single or double quotes). For example: `- {{ MyConstant }}` is not valid, as it is not identified as a string. However, the following references are both valid: `- '{{ MyConstant }}'` and `- "{{ MyConstant }}"`. **Examples** Constant referenced in step inputs ``` name: Download Amazon CLI version 2 schemaVersion: 1.0 constants: - Source: type: string value: https://awscli.amazonaws.com/AWSCLIV2.msi phases: - name: build steps: - name: Download action: WebDownload inputs: - source: '{{ Source }}' destination: 'C:\Windows\Temp\AWSCLIV2.msi' ``` Constant referenced in loop inputs ``` name: PingHosts schemaVersion: 1.0 constants: - Hosts: type: string value: 127.0.0.1,amazon.com phases: - name: build steps: - name: Ping action: ExecuteBash loop: forEach: list: '{{ Hosts }}' delimiter: ',' inputs: commands: - ping -c 4 {{ loop.value }} ``` ## Parameters Parameters are mutable variables, with settings that the calling application can provide at runtime. You can define parameters in the `Parameters` section of the YAML document. **Rules for parameter names** + The name must be between 3 and 128 characters in length. + The name can only contain alphanumeric characters (a-z, A-Z, 0-9), dashes (-), or underscores (\$1). + The name must be unique within the document. + The name must be specified as a YAML string. ### Syntax ``` parameters: - : type: default: description: ``` | Key name | Required | Description | | --- | --- | --- | | `name` | Yes | The name of the parameter. Must be unique for the document (it must not be the same as any other parameter names or constants). | | `type` | Yes | The data type of the parameter. Supported types include: `string`. | | `default` | No | The default value for the parameter. | | `description` | No | Describes the parameter. | ### Reference parameter values in a document You can reference parameters in step or loop inputs inside of your YAML document, as follows: + Parameter references are case-sensitive, and the name must match exactly. + The name must be enclosed within double curly braces `{{` *MyParameter* `}}`. + Spaces are allowed within the curly braces, and are automatically trimmed. For example, all of the following references are valid: `{{ MyParameter }}`, `{{ MyParameter}}`, `{{MyParameter }}`, `{{MyParameter}}` + The reference in the YAML document must be specified as a string (enclosed in single or double quotes). For example: `- {{ MyParameter }}` is not valid, as it is not identified as a string. However, the following references are both valid: `- '{{ MyParameter }}'` and `- "{{ MyParameter }}"`. **Examples** The following examples show how to use parameters in your YAML document: + Refer to a parameter in step inputs: ``` name: Download AWS CLI version 2 schemaVersion: 1.0 parameters: - Source: type: string default: 'https://awscli.amazonaws.com/AWSCLIV2.msi' description: The AWS CLI installer source URL. phases: - name: build steps: - name: Download action: WebDownload inputs: - source: '{{ Source }}' destination: 'C:\Windows\Temp\AWSCLIV2.msi' ``` + Refer to a parameter in loop inputs: ``` name: PingHosts schemaVersion: 1.0 parameters: - Hosts: type: string default: 127.0.0.1,amazon.com description: A comma separated list of hosts to ping. phases: - name: build steps: - name: Ping action: ExecuteBash loop: forEach: list: '{{ Hosts }}' delimiter: ',' inputs: commands: - ping -c 4 {{ loop.value }} ``` ### Override parameters at runtime You can use the `--parameters` option from the Amazon CLI with a key-value pair to set a parameter value at runtime. + Specify the parameter key-value pair as the name and value, separated by an equals sign (=). + Multiple parameters must be separated by a comma. + Parameter names that are not found in the YAML component document are ignored. + The parameter name and value are both required. **Important** Component parameters are plain text values, and are logged in Amazon CloudTrail. We recommend that you use Amazon Secrets Manager or the Amazon Systems Manager Parameter Store to store your secrets. For more information about Secrets Manager, see [What is Secrets Manager?](https://docs.amazonaws.cn/secretsmanager/latest/userguide/intro.html) in the *Amazon Secrets Manager User Guide*. For more information about Amazon Systems Manager Parameter Store, see [Amazon Systems Manager Parameter Store](https://docs.amazonaws.cn/systems-manager/latest/userguide/systems-manager-parameter-store.html) in the *Amazon Systems Manager User Guide*. #### Syntax ``` --parameters name1=value1,name2=value2... ``` | CLI option | Required | Description | | --- | --- | --- | | --parameters *name*=*value*,... | No | This option takes list of key-value pairs, with the parameter name as the key. | **Examples** The following examples show how to use parameters in your YAML document: + The parameter key-value pair specified in this `--parameter` option is not valid: ``` --parameters ntp-server= ``` + Set one parameter key-value pair with the `--parameter` option in the Amazon CLI: ``` --parameters ntp-server=ntp-server-windows-qe.us-east1.amazon.com ``` + Set multiple parameter key-value pairs with the `--parameter` option in the Amazon CLI: ``` --parameters ntp-server=ntp-server.amazon.com,http-url=https://internal-us-east1.amazon.com ``` ## Use Systems Manager Parameter Store parameters You can reference Amazon Systems Manager Parameter Store parameters (SSM parameters) in your component documents by prefixing variables with `aws:ssm`. For example, `{{ aws:ssm:/my/param }}` resolves to the value of the SSM parameter `/my/param`. This feature supports the following SSM parameter types: + String – Maps to the EC2 TOE string type. + StringList – Maps to the EC2 TOE `stringList` type. + SecureString – Maps to the EC2 TOE string type. For more information about the Parameter Store see [Amazon Systems Manager Parameter Store](https://docs.amazonaws.cn/systems-manager/latest/userguide/systems-manager-parameter-store.html) in the *Amazon Systems Manager User Guide*. You can also reference Amazon Secrets Manager secrets using an SSM parameter `SecureString`. For example: `{{ aws:ssm:/aws/reference/secretsmanager/test/test-secret }}`. For more information, see [Referencing Amazon Secrets Manager secrets from Parameter Store parameters](https://docs.amazonaws.cn/systems-manager/latest/userguide/integration-ps-secretsmanager.html). **Important** Image Builder excludes `SecureString` parameter resolution from its logs. However, you are also responsible for ensuring that sensitive information is not logged through commands issued in the component document. For example, if you use the `echo` command with a secure string, the command writes a plaintext value to the log. ### Required IAM permissions To use Systems Manager parameters in your components, your instance role must have the `ssm:GetParameter` permission for the parameter resource ARN. For example: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws-cn:ssm:*:111122223333:parameter/ImageBuilder-*" } ] } ``` ------ To access encrypted values, you'll also need the following permissions: + Add `kms:Decrypt` for `SecureString` parameters or Amazon Secrets Manager values that are encrypted with a customer managed Amazon KMS key. + Add `secretsmanager:GetSecretValue` if you reference a Secrets Manager secret. ### Reference an SSM parameter in a component document The following example shows how to reference an Systems Manager Parameter Store parameter of Systems Manager parameters in a component: ``` name: UseSSMParameterVariable description: This is a sample component document that prints out the value of an SSM Parameter. Never do this for a SecureString parameter. schemaVersion: 1.0 phases: - name: verify steps: - name: EchoParameterValue action: ExecuteBash inputs: commands: - echo "Log SSM parameter name: /my/test/param, value {{ aws:ssm:/my/test/param }}." ``` ### Dynamic runtime variable resolution for SSM parameters AWSTOE provides the following built-in function that you can use within variable references to manipulate or transform values at runtime. #### resolve function The `resolve` function resolves a variable reference inside of another variable reference, allowing for dynamic variable name referencing. This is useful when working with SSM parameters where part of the parameter path may be variable and passed in as a document parameter. The `resolve` function only supports dynamic resolution of the name portion of an SSM parameter. ##### Syntax The `dynamic_variable` in the following example represents the name of an SSM parameter, and must be one of the following: + An SSM parameter reference (for example, `aws:ssm:/my/param`) + A component document parameter reference (for example, `parameter-name`) ``` {{ aws:ssm:resolve(dynamic_variable) }} ``` ##### Example: Resolve an SSM parameter at runtime The following example shows how to use the `resolve` function in a YAML component document: ``` name: SsmParameterTest description: This component verifies an SSM parameter variable reference with the echo command. schemaVersion: 1.0 parameters: - parameter-name: type: string description: "test" phases: - name: validate steps: - name: PrintDynamicVariable action: ExecuteBash inputs: commands: - echo "{{ aws:ssm:resolve(parameter-name) }}" ``` # Use conditional constructs in EC2 TOE Conditional constructs perform different actions in your component document based on whether the specified conditional expression evaluates to `true` or `false`. You can use the `if` construct to control the flow of execution in your component document. ## if Construct You can use the `if` construct to evaluate whether a step should run or not. By default, when the `if` conditional expression evaluates to `true`, EC2 TOE runs the step, and when the condition evaluates to `false`, EC2 TOE skips the step. If a step is skipped, it's treated as a successful step when EC2 TOE evaluates whether the phase and document ran successfully. **Note** An `if` statement is only evaluated one time, even if the step triggers a restart. If a step restarts, it recognizes that the `if` statement has already been evaluated, and continues where it left off. ### Syntax ``` if: - : [then: ] [else: ] ``` | Key name | Required | Description | | --- | --- | --- | | conditional expression | Yes | The conditional expression can contain exactly one of the following types of operators at the top level. [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/imagebuilder/latest/userguide/toe-conditional-constructs.html) If your expression must satisfy multiple conditions, use a logical operator to specify your conditions. | | then | No | Defines the action to take if the conditional expression evaluates to `true`. | | else | No | Defines the action to take if the conditional expression evaluates to `false`. | | step action | Conditional | When you use `then` or `else`, you must specify one of the following step actions: [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/imagebuilder/latest/userguide/toe-conditional-constructs.html) | **Example 1: Install package** The following example steps from an EC2 TOE component document use logical operators to test a parameter value and run the appropriate package manager commands to install an application if the package is unzipped. ``` - name: InstallUnzipAptGet action: ExecuteBash if: and: - binaryExists: 'apt-get' - not: binaryExists: 'unzip' inputs: commands: - sudo apt-get update - sudo apt-get install -y unzip - name: InstallUnzipYum action: ExecuteBash if: and: - binaryExists: 'yum' - not: binaryExists: 'unzip' inputs: commands: - sudo yum install -y unzip - name: InstallUnzipZypper action: ExecuteBash if: and: - binaryExists: 'zypper' - not: binaryExists: 'unzip' inputs: commands: - sudo zypper refresh - sudo zypper install -y unzip ``` **Example 2: Skip a step** The following example shows two ways to skip a step. One uses a logical operator, and one uses a comparison operator with the `Skip` step action. ``` # Creates a file if it does not exist using not - name: CreateMyConfigFile-1 action: ExecuteBash if: not: fileExists: '/etc/my_config' inputs: commands: - echo "Hello world" > '/etc/my_config' # Creates a file if it does not exist using then and else - name: CreateMyConfigFile-2 action: ExecuteBash if: fileExists: '/etc/my_config' then: Skip else: Execute inputs: commands: - echo "Hello world" > '/etc/my_config' ``` # Use comparison operators in EC2 TOE component documents You can use the following comparison operators with the **[Assert](toe-action-modules.md#action-modules-assertion)** action module and with conditional expressions that use the [if ConstructSyntax](toe-conditional-constructs.md#toe-conditional-if). A comparison operator can operate on a single value, for example `stringIsEmpty`, or it can compare a baseline value to a second value (variable value) to determine whether the conditional expression evaluates to `true` or `false`. If the comparison operates on two values, the second value can be a chaining variable. When comparing values of a different type, the following value conversions can occur prior to the comparison: + For numeric comparisons, if the variable value is a string, EC2 TOE converts the string to a number prior to the evaluation. If the conversion is not possible, the comparison returns `false`. For example, if the variable value is `"1.0"`, the conversion works, but if the variable value is `"a10"` the conversion fails. + For string comparisons, if the variable value is a number, EC2 TOE converts it to a string prior to the evaluation. ## Compare strings The following comparison operators work with strings to compare values, to test for spaces or an empty string, or to compare an input value to a regex pattern. String comparisons are not case sensitive, and they don't trim spaces from the beginning or the end of the string inputs. **String comparison operators** + [stringIsEmpty](#stringIsEmpty) + [stringIsWhitespace](#stringIsWhitespace) + [stringEquals](#stringEquals) + [stringLessThan](#stringLessThan) + [stringLessThanEquals](#stringLessThanEquals) + [stringGreaterThan](#stringGreaterThan) + [stringGreaterThanEquals](#stringGreaterThanEquals) + [patternMatches](#patternMatches) **stringIsEmpty** The `stringIsEmpty` operator returns `true` if the specified string doesn't contain any characters. For example: ``` # Evaluates to true stringIsEmpty: "" # Evaluates to false stringIsEmpty: " " # Evaluates to false stringIsEmpty: "Hello." ``` **stringIsWhitespace** Tests if the string specified for `stringIsWhitespace` contains only spaces. For example: ``` # Evaluates to true stringIsWhitespace: " " # Evaluates to false stringIsWhitespace: "" # Evaluates to false stringIsWhitespace: " Hello?" ``` **stringEquals** Tests if the string specified for `stringEquals` is an exact match for the string specified in the `value` parameter. For example: ``` # Evaluates to true stringEquals: 'Testing, testing...' value: 'Testing, testing...' # Evaluates to false stringEquals: 'Testing, testing...' value: 'Hello again.' # Evaluates to false stringEquals: 'Testing, testing...' value: 'TESTING, TESTING....' # Evaluates to false stringEquals: 'Testing, testing...' value: ' Testing, testing...' # Evaluates to false stringEquals: 'Testing, testing...' value: 'Testing, testing... ' ``` **stringLessThan** Tests if the string specified for `stringLessThan` is less than the string specified in the `value` parameter. For example: ``` # Evaluates to true # This comparison operator isn't case sensitive stringlessThan: 'A' value: 'a' # Evaluates to true - 'a' is less than 'b' stringlessThan: 'b' value: 'a' # Evaluates to true # Numeric strings compare as less than alphabetic strings stringlessThan: 'a' value: '0' # Evaluates to false stringlessThan: '0' value: 'a' ``` **stringLessThanEquals** Tests if the string specified for `stringLessThanEquals` is less than or equal to the string specified in the `value` parameter. For example: ``` # Evaluates to true - 'a' is equal to 'a' stringLessThanEquals: 'a' value: 'a' # Evaluates to true - since the comparison isn't case sensitive, 'a' is equal to 'A' stringLessThanEquals: 'A' value: 'a' # Evaluates to true - 'a' is less than 'b' stringLessThanEquals: 'b' value: 'a' # Evaluates to true - '0' is less than 'a' stringLessThanEquals: 'a' value: '0' # Evaluates to false - 'a' is greater than '0' stringLessThanEquals: '0' value: 'a' ``` **stringGreaterThan** Tests if the string specified for `stringGreaterThan` is greater than the string specified in the `value` parameter. For example: ``` # Evaluates to false - since the comparison isn't case sensitive, 'A' is equal to 'a' stringGreaterThan: 'a' value: 'A' # Evaluates to true - 'b' is greater than 'a' stringGreaterThan: 'a' value: 'b' # Evaluates to true - 'a' is greater than '0' stringGreaterThan: '0' value: 'a' # Evaluates to false - '0' is less than 'a' stringGreaterThan: 'a' value: '0' ``` **stringGreaterThanEquals** Tests if the string specified for `stringGreaterThanEquals` is greater than or equal to the string specified in the `value` parameter. For example: ``` # Evaluates to true - 'a' is equal to 'A' stringGreaterThanEquals: 'A' value: 'a' # Evaluates to true - 'b' is greater than 'a' stringGreaterThanEquals: 'a' value: 'b' # Evaluates to true - 'a' is greater than '0' stringGreaterThanEquals: '0' value: 'a' # Evaluates to false - '0' is less than 'a' stringGreaterThanEquals: 'a' value: '0' ``` **patternMatches** Tests if the string specified in the `value` parameter matches the regex pattern specified for `patternMatches`. The comparison uses to the [Golang regexp package](https://pkg.go.dev/regexp), which conforms to the RE2 syntax. For more information about RE2 rules, see the [google / re2](https://github.com/google/re2/wiki/Syntax) repository in *GitHub*. The following example shows a pattern match that returns `true`: ``` patternMatches: '^[a-z]+$' value: 'ThisIsValue' ``` ## Compare numbers The following comparison operators work with numbers. The values provided for these operators must be one of the following types, according to the YAML specification. Support for numeric comparisons uses the golang big package comparison operator, for example: [func (\$1Float) Cmp](https://pkg.go.dev/math/big#Float.Cmp). + Integer + Float (based on float64, which supports numbers from -1.7e\$1308 to \$11.7e\$1308) + A string that matches the following regex pattern: `^[-+]?([0-9]+[.])?[0-9]+$` **Number comparison operators** + [numberEquals](#numberEquals) + [numberLessThan](#numberLessThan) + [numberLessThanEquals](#numberLessThanEquals) + [numberGreaterThan](#numberGreaterThan) + [numberGreaterThanEquals](#numberGreaterThanEquals) **numberEquals** Tests if the number specified for `numberEquals` is equal to the number specified in the `value` parameter. All of the following example comparisons return `true`: ``` # Values provided as a positive number numberEquals: 1 value: 1 # Comparison value provided as a string numberEquals: '1' value: 1 # Value provided as a string numberEquals: 1 value: '1' # Values provided as floats numberEquals: 5.0 value: 5.0 # Values provided as a negative number numberEquals: -1 value: -1 ``` **numberLessThan** Tests if the number specified for `numberLessThan` is less than the number specified in the `value` parameter. For example: ``` # Evaluates to true numberLessThan: 2 value: 1 # Evaluates to true numberLessThan: 2 value: 1.9 # Evaluates to false numberLessThan: 2 value: '2' ``` **numberLessThanEquals** Tests if the number specified for `numberLessThanEquals` is less than or equal to the number specified in the `value` parameter. For example: ``` # Evaluates to true numberLessThanEquals: 2 value: 1 # Evaluates to true numberLessThanEquals: 2 value: 1.9 # Evaluates to true numberLessThanEquals: 2 value: '2' # Evaluates to false numberLessThanEquals: 2 value: 2.1 ``` **numberGreaterThan** Tests if the number specified for `numberGreaterThan` is greater than the number specified in the `value` parameter. For example: ``` # Evaluates to true numberGreaterThan: 1 value: 2 # Evaluates to true numberGreaterThan: 1 value: 1.1 # Evaluates to false numberGreaterThan: 1 value: '1' ``` **numberGreaterThanEquals** Tests if the number specified for `numberGreaterThanEquals` is greater than or equal to the number specified in the `value` parameter. For example: ``` # Evaluates to true numberGreaterThanEquals: 1 value: 2 # Evaluates to true numberGreaterThanEquals: 1 value: 1.1 # Evaluates to true numberGreaterThanEquals: 1 value: '1' # Evaluates to false numberGreaterThanEquals: 1 value: 0.8 ``` ## Check files The following comparison operators check the file hash or check if a file or folder exists. **File and folder operators** + [binaryExists](#binaryExists) + [fileExists](#fileExists) + [folderExists](#folderExists) + [fileMD5Equals](#fileMD5Equals) + [fileSHA1Equals](#fileSHA1Equals) + [fileSHA256Equals](#fileSHA256Equals) + [fileSHA512Equals](#fileSHA512Equals) **binaryExists** Tests whether an application is available in the current path. For example: ``` binaryExists: 'foo' ``` On Linux and macOS systems, for an application named *foo*, this works the same as the following bash command: **type *foo* >/dev/null 2>&1**, where **\$1? == 0** indicates a successful comparison. On Windows systems, for an application named *foo*, this works the same as the PowerShell command **& C:\$1Windows\$1System32\$1where.exe /Q *foo*** where **\$1LASTEXITCODE = 0** indicates a successful comparison. **fileExists** Tests whether a file exists at the specified path. You can provide an absolute or relative path. If the location you specify exists and is a file, the comparison evaluates to `true`. For example: ``` fileExists: '/path/to/file' ``` On Linux and macOS systems, this works the same as the following bash command: **-d */path/to/file***, where **\$1? == 0** indicates a successful comparison. On Windows systems, this works the same as the PowerShell command **Test-Path -Path '*C:\$1path\$1to\$1file*' -PathType 'Leaf'**. **folderExists** Tests whether a folder exists at the specified path. You can provide an absolute or relative path. If the location you specify exists and is a folder, the comparison evaluates to `true`. For example: ``` folderExists: '/path/to/folder' ``` On Linux and macOS systems, this works the same as the following bash command: **-d */path/to/folder***, where **\$1? == 0** indicates a successful comparison. On Windows systems, this works the same as the PowerShell command **Test-Path -Path '*C:\$1path\$1to\$1folder*' -PathType 'Container'**. **fileMD5Equals** Tests whether a file’s MD5 hash equals a specified value. For example: ``` fileMD5Equals: '' path: '/path/to/file' ``` **fileSHA1Equals** Tests whether a file’s SHA1 hash equals a specified value. For example: ``` fileSHA1Equals: '' path: '/path/to/file' ``` **fileSHA256Equals** Tests whether a file’s SHA256 hash equals a specified value. For example: ``` fileSHA256Equals: '' path: '/path/to/file' ``` **fileSHA512Equals** Tests whether a file’s SHA512 hash equals a specified value. For example: ``` fileSHA512Equals: '' path: '/path/to/file' ``` # Use logical operators in EC2 TOE component documents You can use the following logical operators to add or modify conditional expressions in your component document. EC2 TOE evaluates conditional expressions in the order that the conditions are specified. For more information about comparison operators for component documents, see [Use comparison operators in EC2 TOE component documents](toe-comparison-operators.md). **and** With the `and` operator, you can evaluate two or more comparisons as a single expression. The expression evaluates to `true` when all of the conditions in the list are true. Otherwise, the expression evaluates to `false`. **Examples:** The following example performs two comparisons – a string and a number. Both comparisons are true, so the expression evaluates to true. ``` and: - stringEquals: 'test_string' value: 'test_string' - numberEquals: 1 value: 1 ``` The following example also performs two comparisons. The first comparison is false, at which point evaluation stops and the second comparison is skipped. The expression evaluates to `false`. ``` and: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 1 ``` **or** With the `or` operator, you can evaluate two or more comparisons as a single expression. The expression evaluates to `true` when one of the specified comparisons is true. If none of the specified comparisons evaluate to `true`, the expression evaluates to `false`. **Examples:** The following example performs two comparisons – a string and a number. The first comparison is true, so the expression evaluates to `true` and the second comparison is skipped. ``` or: - stringEquals: 'test_string' value: 'test_string' - numberEquals: 1 value: 3 ``` The following example also performs two comparisons. The first comparison is false, and evaluation continues. The second comparison is true, so the expression evaluates to `true`. ``` or: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 1 ``` In the final example, both comparisons are false, so the expression evaluates to `false`. ``` or: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 3 ``` **not** With the `not` operator, you can negate a single comparison. The expression evaluates to `true` if the comparison is false. If the comparison is true, then the expression evaluates to `false`. **Examples:** The following example performs a string comparison. The comparison is false, so the expression evaluates to `true`. ``` not: - stringEquals: 'test_string' value: 'Hello world!' ``` The following example also performs a string comparison. The comparison is true, so the expression evaluates to `false`. ``` not: - stringEquals: 'test_string' value: 'test_string' ``` # Use looping constructs in EC2 TOE This section provides information to help you create looping constructs in the EC2 TOE. Looping constructs define a repeated sequence of instructions. You can use the following types of looping constructs in EC2 TOE: + `for` constructs – Iterate over a bounded sequence of integers. + `forEach` constructs + `forEach` loop with input list – Iterates over a finite collection of strings. + `forEach` loop with delimited list – Iterates over a finite collection of strings joined by a delimiter. **Note** Looping constructs support only string data types. **Topics** + [ ## Reference iteration variables ](#toe-loop-iteration-variables) + [ ## Types of looping constructs ](#toe-loop-types) + [ ## Step fields ](#toe-loop-step-fields) + [ ## Step and iteration outputs ](#toe-loop-step-output) ## Reference iteration variables To refer to the index and value of the current iteration variable, the reference expression `{{ loop.* }}` must be used within the input body of a step that contains a looping construct. This expression cannot be used to refer to the iteration variables of the looping construct of another step. The reference expression consists of the following members: + `{{ loop.index }}` – The ordinal position of the current iteration, which is indexed at `0`. + `{{ loop.value }}` – The value associated with the current iteration variable. ### Loop names All looping constructs have an optional name field for identification. If a loop name is provided, it can be used to refer to iteration variables in the input body of the step. To refer to the iteration indices and values of a named loop, use `{{ .* }}` with `{{ loop.* }}` in the input body of the step. This expression cannot be used to refer to the named looping construct of another step. The reference expression consists of the following members: + `{{ .index }}` – The ordinal position of the current iteration of the named loop, which is indexed at `0`. + `{{ .value }}` – The value associated with the current iteration variable of the named loop. ### Resolve reference expressions The EC2 TOE resolves reference expressions as follows: + `{{ .* }}` – EC2 TOE resolves this expression using the following logic: + If the loop of the currently running step matches the `` value, then the reference expression resolves to the looping construct of the currently running step. + `` resolves to the named looping construct if it appears within the currently running step. + `{{ loop.* }}` – EC2 TOE resolves the expression using the looping construct defined in the currently running step. If reference expressions are used within a step that does not contain a loop, then EC2 TOE does not resolve the expressions and they appear in the step with no replacement. **Note** Reference expressions must be enclosed in double quotes to be correctly interpreted by the YAML compiler. ## Types of looping constructs This section provides information and examples about looping construct types that can be used in the EC2 TOE. **Topics** + [ ### `for` loop ](#toe-loop-types-for) + [ ### `forEach` loop with input list ](#toe-loop-types-foreach) + [ ### `forEach` loop with delimited list ](#toe-loop-types-foreach-delimited) ### `for` loop The `for` loop iterates on a range of integers specified within a boundary outlined by the start and end of the variables. The iterating values are in the set `[start, end]` and includes boundary values. EC2 TOE verifies the `start`, `end`, and `updateBy` values to ensure that the combination does not result in an infinite loop. `for` loop schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" for: start: int end: int updateBy: int inputs: ... ``` **`for` loop input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name of the loop. It must be unique compared to other loop names in the same phase. | String | No | "" | | `start` | Starting value of iteration. Does not accept chaining expressions. | Integer | Yes | n/a | | `end` | Ending value of iteration. Does not accept chaining expressions. | Integer | Yes | n/a | | `updateBy` | Difference by which an iterating value is updated through addition. It must be a negative or positive non-zero value. Does not accept chaining expressions. | Integer | Yes | n/a | `for` loop input example ``` - name: "CalculateFileUploadLatencies" action: "ExecutePowerShell" loop: for: start: 100000 end: 1000000 updateBy: 100000 inputs: commands: - | $f = new-object System.IO.FileStream c:\temp\test{{ loop.index }}.txt, Create, ReadWrite $f.SetLength({{ loop.value }}MB) $f.Close() - c:\users\administrator\downloads\latencyTest.exe --file c:\temp\test{{ loop.index }}.txt - Amazon s3 cp c:\users\administrator\downloads\latencyMetrics.json s3://bucket/latencyMetrics.json - | Remove-Item -Path c:\temp\test{{ loop.index }}.txt Remove-Item -Path c:\users\administrator\downloads\latencyMetrics.json ``` ### `forEach` loop with input list The `forEach` loop iterates on an explicit list of values, which can be strings and chained expressions. `forEach` loop with input list schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" forEach: - "string" inputs: ... ``` **`forEach` loop with input list input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name of the loop. It must be unique compared to other loop names in the same phase. | String | No | "" | | List of strings of `forEach` loop | List of strings for iteration. Accepts chained expressions as strings in the list. Chained expressions must be enclosed by double quotes for the YAML compiler to correctly interpret them. | List of strings | Yes | n/a | `forEach` loop with input list example 1 ``` - name: "ExecuteCustomScripts" action: "ExecuteBash" loop: name: BatchExecLoop forEach: - /tmp/script1.sh - /tmp/script2.sh - /tmp/script3.sh inputs: commands: - echo "Count {{ BatchExecLoop.index }}" - sh "{{ loop.value }}" - | retVal=$? if [ $retVal -ne 0 ]; then echo "Failed" else echo "Passed" fi ``` `forEach` loop with input list example 2 ``` - name: "RunMSIWithDifferentArgs" action: "ExecuteBinary" loop: name: MultiArgLoop forEach: - "ARG1=C:\Users ARG2=1" - "ARG1=C:\Users" - "ARG1=C:\Users ARG3=C:\Users\Administrator\Documents\f1.txt" inputs: commands: path: "c:\users\administrator\downloads\runner.exe" args: - "{{ MultiArgLoop.value }}" ``` `forEach` loop with input list example 3 ``` - name: "DownloadAllBinaries" action: "S3Download" loop: name: MultiArgLoop forEach: - "bin1.exe" - "bin10.exe" - "bin5.exe" inputs: - source: "s3://bucket/{{ loop.value }}" destination: "c:\temp\{{ loop.value }}" ``` ### `forEach` loop with delimited list The loop iterates over a string containing values separated by a delimiter. To iterate over the string’s constituents, EC2 TOE uses the delimiter to split the string into an array suitable for iteration. `forEach` loop with delimited list schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" forEach: list: "string" delimiter: ".,;:\n\t -_" inputs: ... ``` **`forEach` loop with delimited list input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name given to the loop. It should be unique when compared to other loop names in the same phase. | String | No | "" | | `list` | A string that is composed of constituent strings joined by a common delimiter character. Also accepts chained expressions. In case of chained expressions, ensure that those are enclosed by double quotes for correct interpretation by the YAML compiler. | String | Yes | n/a | | `delimiter` | Character used to separate out strings within a block. Default is the comma character. Only one delimiter character is allowed from the given list: [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/imagebuilder/latest/userguide/toe-looping-constructs.html) Chaining expressions cannot be used. | String | No | Comma: "," | **Note** The value of `list` is treated as an immutable string. If the source of `list` is changed during runtime, it will not be reflected during the run. `forEach` loop with delimited list example 1 This example uses the following chaining expression pattern to refer to another step's output: `..[inputs | outputs].`. ``` - name: "RunMSIs" action: "ExecuteBinary" loop: forEach: list: "{{ build.GetAllMSIPathsForInstallation.outputs.stdout }}" delimiter: "\n" inputs: commands: path: "{{ loop.value }}" ``` `forEach` loop with delimited list example 2 ``` - name: "UploadMetricFiles" action: "S3Upload" loop: forEach: list: "/tmp/m1.txt,/tmp/m2.txt,/tmp/m3.txt,..." inputs: commands: - source: "{{ loop.value }}" destination: "s3://bucket/key/{{ loop.value }}" ``` ## Step fields Loops are part of a step. Any field related to the running of a step is not applied to individual iterations. Step fields apply only at the step level, as follows: + *timeoutSeconds* – All iterations of the loop must be run within the time period specified by this field. If the loop run times out, then EC2 TOE runs the retry policy of the step and resets the timeout parameter for each new attempt. If the loop run exceeds the timeout value after reaching the maximum number of retries, the failure message of the step states that the loop run had timed out. + *onFailure* – Failure handling is applied to the step as follows: + If *onFailure* is set to `Abort`, EC2 TOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, EC2 TOE marks the current step as failed, and stops running the process. EC2 TOE sets the status code for the parent phase and document to `Failed`. **Note** No further steps run after the failed step. + If *onFailure* is set to `Continue`, EC2 TOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, EC2 TOE marks the current step as failed, and continues on to run the next step. EC2 TOE sets the status code for the parent phase and document to `Failed`. + If *onFailure* is set to `Ignore`, EC2 TOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, EC2 TOE marks the current step as `IgnoredFailure`, and continues on to run the next step. EC2 TOE sets the status code for the parent phase and document to `SuccessWithIgnoredFailure`. **Note** This is still considered a successful run, but includes information to let you know that one or more steps failed and were ignored. + *maxAttempts * – For every retry, the entire step and all iterations are run from the beginning. + *status* – The overall status of the running of a step.`status` does not represent the status of individual iterations. The status of a step with loops is determined as follows: + If a single iteration fails to run, the status of a step points to a failure. + If all iterations succeed, the status of a step points to a success. + *startTime * – The overall start time of the running of a step. Does not represent the start time of individual iterations. + *endTime * – The overall end time of the running of a step. Does not represent the end time of individual iterations. + *failureMessage * – Includes the iteration indices that failed in case of non-timeout errors. In case of timeout errors, the message states that the loop run has failed. Individual error messages for each iteration are not provided to minimize the size of failure messages. ## Step and iteration outputs Every iteration contains an output. At the end of a loop run, EC2 TOE consolidates all successful iteration outputs in `detailedOutput.json`. The consolidated outputs are a collation of values that belong to the corresponding output keys as defined in the output schema of the action module. The following example shows how the outputs are consolidated: **Output of `ExecuteBash` for Iteration 1** ``` { "stdout":"Hello" } ``` **Output of `ExecuteBash` for Iteration 2** ``` { "stdout":"World" } ``` **Output of `ExecuteBash` for Step** ``` { "stdout":"Hello\nWorld" } ``` For example, `ExecuteBash`, `ExecutePowerShell`, and `ExecuteBinary` are action modules which return `STDOUT` as the action module output. `STDOUT` messages are joined with the new line character to produce the overall output of the step in `detailedOutput.json`. EC2 TOE will not consolidate the outputs of unsuccessful iterations.