Building a custom Amazon ParallelCluster AMI

When using the Amazon ParallelCluster command line interface (CLI) or API, you only pay for the Amazon resources that are created when you create or update Amazon ParallelCluster images and clusters. For more information, see Amazon services used by Amazon ParallelCluster.

The Amazon ParallelCluster UI is built on a serverless architecture and you can use it within the Amazon Free Tier category for most cases. For more information, see Amazon ParallelCluster UI costs.

Before reading further, we recommend that you first review the Custom bootstrap actions section. Determine if the modifications that you want to make can be scripted and supported with future Amazon ParallelCluster releases.

Even though building a custom AMI in general isn't ideal, there are specific scenarios where building a custom AMI for Amazon ParallelCluster is necessary. This tutorial covers how to build a custom AMI for these scenarios.

How to customize the Amazon ParallelCluster AMI

There are two ways to build a custom Amazon ParallelCluster AMI. One of these two methods is to build a new AMI using the Amazon ParallelCluster CLI. Another method requires that you to make manual modifications to build a new AMI that's available under your Amazon Web Services account.

Build a custom Amazon ParallelCluster AMI

If you have a customized AMI and software, you can apply the changes that are needed by Amazon ParallelCluster on top of it. Amazon ParallelCluster relies on the EC2 Image Builder service to build customized AMIs. For more information, see the Image Builder User Guide.

Key points:

Steps:

Troubleshooting and monitoring AMI creation process

Image creation completes in about an hour. You can monitor the process by running the pcluster describe-image command or log retrieval commands.

$ pcluster describe-image --image-id IMAGE_ID --region REGION

The build-image command creates a CloudFormation stack with all the EC2 resources that are required to build the image, and launches the EC2 Image Builder process.

After running the build-image command, it's possible to retrieve CloudFormation stack events by using pcluster get-image-stack-events. You can filter results with the --query parameter to see the latest events. For more information, see Filtering Amazon CLI output in the Amazon Command Line Interface User Guide.

$ pcluster get-image-stack-events --image-id IMAGE_ID --region REGION --query "events[0]"
{
 "eventId": "ParallelClusterImage-CREATE_IN_PROGRESS-2022-04-05T21:39:24.725Z",
 "physicalResourceId": "arn:aws:imagebuilder:us-east-1:123456789012:image/parallelclusterimage-IMAGE_ID/3.7.0/1",
 "resourceStatus": "CREATE_IN_PROGRESS",
 "resourceStatusReason": "Resource creation Initiated",
 "resourceProperties": "{\"InfrastructureConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:infrastructure-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"ImageRecipeArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:image-recipe/parallelclusterimage-IMAGE_ID/3.7.0\",\"DistributionConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"Tags\":{\"parallelcluster:image_name\":\"IMAGE_ID\",\"parallelcluster:image_id\":\"IMAGE_ID\"}}",
 "stackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/IMAGE_ID/abcd1234-ef56-gh78-ij90-1234abcd5678",
 "stackName": "IMAGE_ID",
 "logicalResourceId": "ParallelClusterImage",
 "resourceType": "AWS::ImageBuilder::Image",
 "timestamp": "2022-04-05T21:39:24.725Z"
}

After about 15 minutes, the stack events appear in the log event entry related to Image Builder creation. You can now list image log streams and monitor the Image Builder steps by using pcluster list-image-log-streams and pcluster get-image-log-events commands.

$ pcluster list-image-log-streams --image-id IMAGE_ID --region REGION \
    --query 'logStreams[*].logStreamName'

 "3.7.0/1"
]

$ pcluster get-image-log-events --image-id IMAGE_ID --region REGION \
--log-stream-name 3.7.0/1 --limit 3
{
 "nextToken": "f/36295977202298886557255241372854078762600452615936671762",
 "prevToken": "b/36295977196879805474012299949460899222346900769983430672",
 "events": [
   {
     "message": "ExecuteBash: FINISHED EXECUTION",
     "timestamp": "2022-04-05T22:13:26.633Z"
   },
   {
     "message": "Document arn:aws:imagebuilder:us-east-1:123456789012:component/parallelclusterimage-test-abcd1234-ef56-gh78-ij90-1234abcd5678/3.7.0/1",
     "timestamp": "2022-04-05T22:13:26.741Z"
   },
   {
     "message": "TOE has completed execution successfully",
     "timestamp": "2022-04-05T22:13:26.819Z"
   }
 ]
}

Continue to check with the describe-image command until you see the BUILD_COMPLETE status.

$ pcluster describe-image --image-id IMAGE_ID --region REGION
{
 "imageConfiguration": {
   "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.us-east-1.amazonaws.com/parallelcluster/3.7.0/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?Signature=..."
 },
 "imageId": "IMAGE_ID",
 "imageBuildStatus": "BUILD_COMPLETE",
 "region": "us-east-1",
 "ec2AmiInfo": {
     "amiName": "IMAGE_ID 2022-04-05T21-39-24.020Z",
     "amiId": "ami-1234stuv5678wxyz",
     "description": "Amazon ParallelCluster AMI for alinux2, kernel-4.14.238-182.422.amzn2.x86_64, lustre-2.10.8-5.amzn2.x86_64, efa-1.13.0-1.amzn2.x86_64, dcv-2021.1.10598-1.el7.x86_64, slurm-20-11-8-1",
     "state": "AVAILABLE",
     "tags": [
      {
        "value": "2021.3.11591-1.el7.x86_64",
        "key": "parallelcluster:dcv_version"
      },
      ...
     ],
   "architecture": "x86_64"
 },
 "version": "3.7.0"      
}

If you need to troubleshoot a custom AMI creation issue, create an archive of the image logs as described in following steps.

It's possible to archive the logs in an Amazon S3 bucket or in a local file, depending on the --output parameter.

$ pcluster export-image-logs --image-id IMAGE_ID --region REGION \
--bucket BUCKET_NAME --bucket-prefix BUCKET_FOLDER
{
 "url": "https://BUCKET_NAME.s3.us-east-1.amazonaws.com/BUCKET-FOLDER/IMAGE_ID-logs-202209071136.tar.gz?AWSAccessKeyId=..."
}

$ pcluster export-image-logs --image-id IMAGE_ID \
--region REGION --bucket BUCKET_NAME --bucket-prefix BUCKET_FOLDER --output-file /tmp/archive.tar.gz
{
 "path": "/tmp/archive.tar.gz"
}

The archive contains the CloudWatch Logs Streams related to the Image Builder process and Amazon CloudFormation stack events. The command might take several minutes to run.

Managing Custom AMIs

Starting from Amazon ParallelCluster 3.0.0, a new set of commands has been added in the CLI to build, monitor, and manage the image lifecycle. For more information about the commands, see pcluster commands.

Modify an Amazon ParallelCluster AMI

This method consists of modifying an official Amazon ParallelCluster AMI by adding customization on top of it. The base Amazon ParallelCluster AMIs are updated with new releases. These AMIs have all of the components that are required for Amazon ParallelCluster to function when it's installed and configured. You can start with one of these as your base.

Key points:

Steps: