Amazon ParallelCluster API

What is Amazon ParallelCluster API?

Amazon ParallelCluster API is a serverless application that, once deployed to your Amazon Web Services account, provides programmatic access to Amazon ParallelCluster features through an API.

Amazon ParallelCluster API is distributed as a self-contained Amazon CloudFormation template that includes an Amazon API Gateway endpoint, that exposes Amazon ParallelCluster features, and an Amazon Lambda function, that takes care of processing the invoked features.

The following image shows a high-level architecture diagram of the Amazon ParallelCluster API infrastructure.

Amazon ParallelCluster API Documentation

The OpenAPI specification file describing the Amazon ParallelCluster API can be downloaded from:

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/ParallelCluster.openapi.yaml

Starting from the OpenAPI specification file, you can generate documentation for the Amazon ParallelCluster API by using one of the many available tools such as Swagger UI or Redoc.

How to deploy Amazon ParallelCluster API

To deploy Amazon ParallelCluster API you need to be an Administrator of the Amazon Web Services account.

The template used to deploy the API is available at the following URL:

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/parallelcluster-api.yaml

where <REGION> is the Amazon Web Services Region where the API needs to be deployed to and <VERSION> is the Amazon ParallelCluster version (e.g. 3.7.0).

Amazon Lambda processes the API invoked features by using a Lambda layer interface with the Amazon ParallelCluster Python library API.

Deploy with Amazon CLI

Configure Amazon Credentials to be used with the CLI if you haven't already done so.

$ aws configure

Run the following commands to deploy the API:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>  # This can be any name
$ VERSION=3.7.0
$ aws cloudformation create-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \
    --template-url https://${REGION}-aws-parallelcluster.s3.${REGION}.amazonaws.com/parallelcluster/${VERSION}/api/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
$ aws cloudformation wait stack-create-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Customize your deployment

You can customize the API deployment by using the Amazon CloudFormation parameters exposed by the template. To configure the value of a parameter when deploying through the CLI, the following option can be used: --parameters ParameterKey=KeyName,ParameterValue=Value.

The following parameters optional:

Updating the API

Upgrading to a newer Amazon ParallelCluster version

Option 1: Remove the existing API by deleting the corresponding Amazon CloudFormation stack and deploying the new API as shown above.

Option 2: Update the existing API by running the following commands:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>  # This needs to correspond to the existing API stack name
$ VERSION=3.7.0
$ aws cloudformation update-stack \
    --region ${REGION} \
    --stack-name ${API_STACK_NAME} \
    --template-url https://${REGION}-aws-parallelcluster.s3.${REGION}.amazonaws.com/parallelcluster/${VERSION}/api/parallelcluster-api.yaml \
    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
$ aws cloudformation wait stack-update-complete --stack-name ${API_STACK_NAME} --region ${REGION}

Invoking Amazon ParallelCluster API

The Amazon ParallelCluster Amazon API Gateway endpoint is configured with AWS_IAM authorization type, and requires all requests to be SigV4 signed with valid IAM credentials (API reference: making http requests).

When deployed with default settings, API invoke permissions are only granted to the default IAM user created with the API.

To retrieve the ARN of the default IAM user, run:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterApiUserRole'].OutputValue" --output text

To obtain temporary credentials for the default IAM user, run the STS AssumeRole command.

You can retrieve the Amazon ParallelCluster API endpoint by running the following command:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterApiInvokeUrl'].OutputValue" --output text

The Amazon ParallelCluster API can be invoked by any HTTP client that complies with the OpenAPI specifications that can be found here:

https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/ParallelCluster.openapi.yaml

Requests need to be SigV4 signed as documented here.

At this time, we do not offer any official API client implementation. However, API clients can be easily generated from the OpenAPI model by using OpenAPI Generator. Once the client is generated SigV4 signing needs to be added if not provided out of the box.

A reference implementation for a Python API client can be found in the Amazon ParallelCluster repository. To learn more about how you can use the Python API client, see the Using the Amazon ParallelCluster API tutorial.

To implement more advanced access control mechanisms, such as Amazon Cognito or Lambda Authorizers, or to further protect the API with Amazon WAF or API keys, follow the Amazon API Gateway documentation.

- !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/parallelcluster*
- !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/parallelcluster/*
- !Sub arn:${AWS::Partition}:iam::aws:policy/CloudWatchAgentServerPolicy
- !Sub arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore
- !Sub arn:${AWS::Partition}:iam::aws:policy/AWSBatchFullAccess
- !Sub arn:${AWS::Partition}:iam::aws:policy/AmazonS3ReadOnlyAccess
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSBatchServiceRole
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole
- !Sub arn:${AWS::Partition}:iam::aws:policy/EC2InstanceProfileForImageBuilder
- !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Accessing the API logs and metrics

API logs are published to Amazon CloudWatch with a retention of 30 days. To retrieve the LogGroup name associated with an API deployment, run the following command:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterLambdaLogGroup'].OutputValue" --output text

Lambda metrics, logs and Amazon X-Ray trace logs can be also accessed through the Lambda console. To retrieve the ARN of the Lambda function associated with an API deployment run the following command:

$ REGION=<region>
$ API_STACK_NAME=<stack-name>
$ aws cloudformation describe-stacks --region ${REGION} --stack-name ${API_STACK_NAME} --query "Stacks[0].Outputs[?OutputKey=='ParallelClusterLambdaArn'].OutputValue" --output text