Using CloudWatch Logs with Amazon Batch - Amazon Batch
Add a CloudWatch Logs IAM policyInstall and configure the CloudWatch agentView CloudWatch Logs
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Using CloudWatch Logs with Amazon Batch

You can configure your Amazon Batch jobs on EC2 resources to send detailed log information and metrics to CloudWatch Logs. Doing this, you can view different logs from your jobs in one convenient location. For more information about CloudWatch Logs, see What is Amazon CloudWatch Logs? in the Amazon CloudWatch User Guide.

Note

By default, CloudWatch Logs is turned on for Amazon Fargate containers.

To turn on and customize CloudWatch Logs logging, review the following one-time configuration tasks:

  • For Amazon Batch compute environments that are based on EC2 resources, add an IAM policy to the ecsInstanceRole role. For more information, see Add a CloudWatch Logs IAM policy.

  • Create an Amazon EC2 launch template that includes detailed CloudWatch monitoring, then specify the template when you create your Amazon Batch compute environment. You can also install the CloudWatch agent on an existing image and then specify the image in the Amazon Batch first-run wizard.

  • (Optional) Configure the awslogs driver. You can add parameters that change the default behavior on both EC2 and Fargate resources. For more information, see Using the awslogs log driver.

Add a CloudWatch Logs IAM policy

Before your jobs can send log data and detailed metrics to CloudWatch Logs, you must create an IAM policy that uses the CloudWatch Logs APIs. After you create the IAM policy, attach it to the ecsInstanceRole role.

Note

If the ECS-CloudWatchLogs policy isn't attached to the ecsInstanceRole role, basic metrics can still be sent to CloudWatch Logs. However, the basic metrics don't include log data or detailed metrics such as free disk space.

Amazon Batch compute environments use Amazon EC2 resources. When you create a compute environment using the Amazon Batch first-run wizard, Amazon Batch creates the ecsInstanceRole role and configures the environment with it.

If you aren't using the first-run wizard, you can specify the ecsInstanceRole role when you create a compute environment in the Amazon Command Line Interface or Amazon Batch API. For more information, see the Amazon CLI Command Reference or Amazon Batch API Reference.

To create the ECS-CloudWatchLogs IAM policy

  1. Open the IAM console at https://console.amazonaws.cn/iam/.

  2. In the navigation pane, choose Policies.

  3. Choose Create policy.

  4. Choose JSON, then enter the following policy:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws-cn:logs:*:*:*"
            ]
        }
    ]
}

  5. Choose Next: Tags.

  6. (Optional) For Add tags, choose Add tag to add a tag to the policy.

  7. Choose Next: Review.

  8. On the Review policy page, for Name, enter ECS-CloudWatchLogs, and then enter an optional Description.

  9. Choose Create policy.

To attach the ECS-CloudWatchLogs policy to ecsInstanceRole

  1. Open the IAM console at https://console.amazonaws.cn/iam/.

  2. In the navigation pane, choose Roles.

  3. Choose ecsInstanceRole. If the role doesn't exist, follow the procedures in Amazon ECS instance role to create the role.

  4. Choose Add Permissions, then choose Attach policies.

  5. Choose the ECS-CloudWatchLogs policy and then choose Attach policy.

Install and configure the CloudWatch agent

You can create an Amazon EC2 launch template that includes CloudWatch monitoring. For more information, see Launch an instance from a launch template and Advanced details in the Amazon EC2 User Guide for Linux Instances.

You can also install the CloudWatch agent on an existing Amazon EC2 AMI and then specify the image in the Amazon Batch first-run wizard. For more information, see Installing the CloudWatch agent and Getting Started with Amazon Batch.

Note

Launch templates are not supported on Amazon Fargate resources.

View CloudWatch Logs

You can view and search CloudWatch Logs logs in the Amazon Web Services Management Console.

Note

It might take a few minutes for data to display in CloudWatch Logs.

To view your CloudWatch Logs data

  1. Open the CloudWatch console at https://console.amazonaws.cn/cloudwatch/.

  2. In the left navigation pane, choose Logs, then choose Log groups.

    CloudWatch console log groups

  3. Choose a log group to view.

    CloudWatch console log streams

  4. Choose a log stream to view. By default, the streams are identified by the first 200 characters of the job name and the Amazon ECS task ID.

    Tip

    To download log stream data, choose Actions.

    CloudWatch console log events