Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Manual set up to develop custom components with EC2 TOE

The EC2 Task Orchestrator and Executor (EC2 TOE) application is a standalone application that creates, validates, and runs commands within a component definition framework. Amazon services can use EC2 TOE to orchestrate workflows, install software, modify system configurations, and test image builds.

Follow these steps to manually install the EC2 TOE application and use it as a stand-alone application to develop custom components. Image Builder takes care of these steps for you, if you use the Image Builder console or Amazon CLI commands to create custom components. For more information, see Create custom components with Image Builder.

Step 1: Install EC2 TOE

To develop components locally, download and install the EC2 TOE application.

Step 2: Set Amazon credentials

EC2 TOE requires Amazon credentials to connect to other Amazon Web Services services, such as Amazon S3 and Amazon CloudWatch, when running tasks, such as:

If you are running EC2 TOE on an EC2 instance, then running EC2 TOE uses the same permissions as the IAM role attached to the EC2 instance.

For more information about IAM roles for EC2, see IAM roles for Amazon EC2.

The following examples show how to set Amazon credentials using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

To set these variables on Linux, macOS, or Unix, use export.

export AWS_ACCESS_KEY_ID=your_access_key_id

export AWS_SECRET_ACCESS_KEY=your_secret_access_key

To set these variables on Windows using PowerShell, use $env.

$env:AWS_ACCESS_KEY_ID=your_access_key_id

$env:AWS_SECRET_ACCESS_KEY=your_secret_access_key

To set these variables on Windows using the command prompt, use set.

set AWS_ACCESS_KEY_ID=your_access_key_id

set AWS_SECRET_ACCESS_KEY=your_secret_access_key

Step 3: Develop component documents locally

Components are authored with plaintext YAML documents. For more information about document syntax, see Use the EC2 TOE component document framework for custom components.

The following are example Hello World component documents to help you get started.

Linux

Some of the Linux component examples in this guide refer to a component document file named hello-world-linux.yml. You can use the following document to get started with those examples.

name: Hello World
description: This is hello world testing document for Linux.
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the build phase.'
  - name: validate
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the validate phase.'
  - name: test
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the test phase.'

Windows

Some of the Windows component examples in this guide refer to a component document file named hello-world-windows.yml. You can use the following document to get started with those examples.

name: Hello World
description: This is Hello World testing document for Windows.
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: HelloWorldStep
        action: ExecutePowerShell
        inputs:
          commands:
            - Write-Host 'Hello World from the build phase.'
  - name: validate
    steps:
      - name: HelloWorldStep
        action: ExecutePowerShell
        inputs:
          commands:
            - Write-Host 'Hello World from the validate phase.'
  - name: test
    steps:
      - name: HelloWorldStep
        action: ExecutePowerShell
        inputs:
          commands:
            - Write-Host 'Hello World from the test phase.'

macOS

Some of the macOS component examples in this guide refer to a component document file named hello-world-macos.yml. You can use the following document to get started with those examples.

name: Hello World
description: This is hello world testing document for macOS.
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the build phase.'
  - name: validate
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the validate phase.'
  - name: test
    steps:
      - name: HelloWorldStep
        action: ExecuteBash
        inputs:
          commands:
            - echo 'Hello World from the test phase.'

Step 4: Validate EC2 TOE components

You can validate the syntax of EC2 TOE components locally with the EC2 TOE application. The following examples show the EC2 TOE application validate command to validate the syntax of a component without running it.

Linux or macOS

awstoe validate --documents /home/user/hello-world.yml

Windows

awstoe.exe validate --documents C:\Users\user\Documents\hello-world.yml

Step 5: Run EC2 TOE components

The EC2 TOE application can run one or more phases of specified documents using the --phases command line argument. Supported values for --phases are build, validate, and test. Multiple phase values can be entered as comma separated values.

When you provide a list of phases, the EC2 TOE application sequentially runs the specified phases of each document. For example, EC2 TOE runs the build and validate phases of document1.yaml, followed by the build and validate phases of document2.yaml.

To ensure that your logs are stored securely and retained for troubleshooting, we recommend configuring log storage in Amazon S3. In Image Builder, the Amazon S3 location for publishing logs is specified in the infrastructure configuration. For more information about infrastructure configuration, see Manage Image Builder infrastructure configuration

If a list of phases is not provided, the EC2 TOE application runs all phases in the order listed in the YAML document.

To run specific phases in single or multiple documents, use the following commands.

Single phase

awstoe run --documents hello-world.yml --phases build

Multiple phases

awstoe run --documents hello-world.yml --phases build,test

Document run

Run all phases in a single document

awstoe run --documents documentName.yaml

Run all phases in multiple documents

awstoe run --documents documentName1.yaml,documentName2.yaml

Enter Amazon S3 information to upload EC2 TOE logs from a user-defined local path (recommended)

awstoe run --documents documentName.yaml --log-s3-bucket-name amzn-s3-demo-destination-bucket --log-s3-key-prefix S3KeyPrefix --log-s3-bucket-owner S3BucketOwner --log-directory local_path

Run all phases in a single document, and display all logs on the console

awstoe run --documents documentName.yaml --trace

Example command

awstoe run --documents s3://bucket/key/doc.yaml --phases build,validate

Run document with unique ID

awstoe run --documents documentName.yaml --execution-id user-provided-id --phases build,test

Get help with EC2 TOE

awstoe --help