本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。
使用 Amazon ParallelCluster API
在本教程中,您将使用 Amazon API Gateway 和 Amazon ParallelCluster CloudFormation 模板API进行构建和测试。然后,您可以使用上提供的示例客户端 GitHub 来使用API。有关使用的更多信息API,请参阅Amazon ParallelCluster API。
本教程摘自HPC面向公共部门客户研讨会
使用 Amazon ParallelCluster 命令行界面 (CLI) 或时API,您只需为创建或更新 Amazon ParallelCluster 映像和集群时创建的 Amazon 资源付费。有关更多信息,请参阅 Amazon 使用的服务 Amazon ParallelCluster。
Amazon ParallelCluster 用户界面基于无服务器架构构建,在大多数情况下,您可以在 Amazon 免费套餐类别中使用它。有关更多信息,请参阅 Amazon ParallelCluster 用户界面成本。
先决条件
-
已在您的计算环境中安装和配置。 Amazon CLI
-
Amazon ParallelCluster 安装在虚拟环境中。有关更多信息,请参阅Amazon ParallelCluster 在虚拟环境中安装。
-
你有一个 A mazon EC2 密钥 pair。
留在您的主用户目录中并激活您的虚拟环境:
-
安装有用的JSON命令行处理器。
$
sudo yum groupinstall -y "Development Tools" sudo yum install -y jq python3-devel -
运行以下命令获取您的 Amazon ParallelCluster 版本并将其分配给环境变量。
$
PCLUSTER_VERSION=$(pcluster version | jq -r '.version') echo "export PCLUSTER_VERSION=${PCLUSTER_VERSION}" |tee -a ~/.bashrc -
创建环境变量并将您的区域 ID 分配给该变量。
$
export AWS_DEFAULT_REGION="us-east-1"
echo "export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}" |tee -a ~/.bashrc -
运行以下命令来部署API。
API_STACK_NAME=
"pc-api-stack"
echo "export API_STACK_NAME=${API_STACK_NAME}" |tee -a ~/.bashrcaws cloudformation create-stack \ --region ${AWS_DEFAULT_REGION} \ --stack-name ${API_STACK_NAME} \ --template-url https://${AWS_DEFAULT_REGION}-aws-parallelcluster.s3.${AWS_DEFAULT_REGION}.amazonaws.com/parallelcluster/${PCLUSTER_VERSION}/api/parallelcluster-api.yaml \ --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \ --parameters ParameterKey=EnableIamAdminAccess,ParameterValue=true
{ "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/my-api-stack/abcd1234-ef56-gh78-ei90-1234abcd5678" }
过程完成后,继续执行下一步。
-
登录到 Amazon Web Services Management Console。
-
选择您的API部署。
-
选择阶段,然后选择一个阶段。
-
URL请注意 API Gateway 提供的用于访问或调用您的API。它以蓝色突出显示。
-
选择资源,然后选择
/clusters
下面的GET
。 -
选择TEST图标,然后向下滚动并选择TEST图标。
将显示
/clusters GET
的响应。
将 Amazon ParallelCluster 源代码克隆cd
到api
目录中,然后安装 Python 客户端库。
-
$
git clone -b v${PCLUSTER_VERSION} https://github.com/aws/aws-parallelcluster aws-parallelcluster-v${PCLUSTER_VERSION} cd aws-parallelcluster-v${PCLUSTER_VERSION}/api$
pip3 install client/src -
导航回您的主用户目录。
-
导出客户端在运行时使用的API网关库URL。
$
export PCLUSTER_API_URL=$( aws cloudformation describe-stacks --stack-name ${API_STACK_NAME} --query 'Stacks[0].Outputs[?OutputKey==`ParallelClusterApiInvokeUrl`].OutputValue' --output text ) echo "export PCLUSTER_API_URL=${PCLUSTER_API_URL}" |tee -a ~/.bashrc -
导出客户端用于创建集群的集群名称。
$
export CLUSTER_NAME="test-api-cluster" echo "export CLUSTER_NAME=${CLUSTER_NAME}" |tee -a ~/.bashrc -
运行以下命令来存储示例客户端用于访问的凭据API。
$
export PCLUSTER_API_USER_ROLE=$( aws cloudformation describe-stacks --stack-name ${API_STACK_NAME} --query 'Stacks[0].Outputs[?OutputKey==`ParallelClusterApiUserRole`].OutputValue' --output text ) echo "export PCLUSTER_API_USER_ROLE=${PCLUSTER_API_USER_ROLE}" |tee -a ~/.bashrc
-
将以下示例客户端代码复制到您的主用户目录中的
test_pcluster_client.py
。客户端代码发出执行以下操作的请求:-
创建集群。
-
描述集群。
-
列出集群。
-
描述计算实例集。
-
描述集群实例。
# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: MIT-0 # # Permission is hereby granted, free of charge, to any person obtaining a copy of this # software and associated documentation files (the "Software"), to deal in the Software # without restriction, including without limitation the rights to use, copy, modify, # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # # Author: Evan F. Bollig (Github: bollig) import time, datetime import os import pcluster_client from pprint import pprint from pcluster_client.api import ( cluster_compute_fleet_api, cluster_instances_api, cluster_operations_api ) from pcluster_client.model.create_cluster_request_content import CreateClusterRequestContent from pcluster_client.model.cluster_status import ClusterStatus region=os.environ.get("AWS_DEFAULT_REGION") # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = pcluster_client.Configuration( host = os.environ.get("PCLUSTER_API_URL") ) cluster_name=os.environ.get("CLUSTER_NAME") # Enter a context with an instance of the API client with pcluster_client.ApiClient(configuration) as api_client: cluster_ops = cluster_operations_api.ClusterOperationsApi(api_client) fleet_ops = cluster_compute_fleet_api.ClusterComputeFleetApi(api_client) instance_ops = cluster_instances_api.ClusterInstancesApi(api_client) # Create cluster build_done = False try: with open('cluster-config.yaml', encoding="utf-8") as f: body = CreateClusterRequestContent(cluster_name=cluster_name, cluster_configuration=f.read()) api_response = cluster_ops.create_cluster(body, region=region) except pcluster_client.ApiException as e: print("Exception when calling create_cluster: %s\n" % e) build_done = True time.sleep(60) # Confirm cluster status with describe_cluster while not build_done: try: api_response = cluster_ops.describe_cluster(cluster_name, region=region) pprint(api_response) if api_response.cluster_status == ClusterStatus('CREATE_IN_PROGRESS'): print('. . . working . . .', end='', flush=True) time.sleep(60) elif api_response.cluster_status == ClusterStatus('CREATE_COMPLETE'): print('READY!') build_done = True else: print('ERROR!!!!') build_done = True except pcluster_client.ApiException as e: print("Exception when calling describe_cluster: %s\n" % e) # List clusters try: api_response = cluster_ops.list_clusters(region=region) pprint(api_response) except pcluster_client.ApiException as e: print("Exception when calling list_clusters: %s\n" % e) # DescribeComputeFleet try: api_response = fleet_ops.describe_compute_fleet(cluster_name, region=region) pprint(api_response) except pcluster_client.ApiException as e: print("Exception when calling compute fleet: %s\n" % e) # DescribeClusterInstances try: api_response = instance_ops.describe_cluster_instances(cluster_name, region=region) pprint(api_response) except pcluster_client.ApiException as e: print("Exception when calling describe_cluster_instances: %s\n" % e)
-
-
创建集群配置。
$
pcluster configure --config cluster-config.yaml -
API客户端库会自动从您的环境变量(例如
AWS_ACCESS_KEY_ID
、或AWS_SESSION_TOKEN
)中检测配置详细信息AWS_SECRET_ACCESS_KEY
,或$HOME/.aws
。以下命令将您当前的IAM角色切换到指定的角色 ParallelClusterApiUserRole.$
eval $(aws sts assume-role --role-arn ${PCLUSTER_API_USER_ROLE} --role-session-name ApiTestSession | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"')需要注意的错误:
如果您看到类似于以下内容的错误,则您已经假设了 ParallelClusterApiUserRole 而且你的
AWS_SESSION_TOKEN
已经过期。An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/ParallelClusterApiUserRole-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/ApiTestSession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/ParallelClusterApiUserRole-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
删除该角色,然后重新运行
aws sts assume-role
命令以使用 ParallelClusterApiUserRole.$
unset AWS_SESSION_TOKEN unset AWS_SECRET_ACCESS_KEY unset AWS_ACCESS_KEY_ID要向您当前的用户提供API访问权限,必须展开资源策略。
-
运行以下命令以测试示例客户端。
$
python3 test_pcluster_client.py{'cluster_configuration': 'Region: us-east-1\n' 'Image:\n' ' Os: alinux2\n' 'HeadNode:\n' ' InstanceType: t2.micro\n' ' Networking . . . :\n' ' SubnetId: subnet-1234567890abcdef0\n' ' Ssh:\n' ' KeyName: adpc\n' 'Scheduling:\n' ' Scheduler: slurm\n' ' SlurmQueues:\n' ' - Name: queue1\n' ' ComputeResources:\n' ' - Name: t2micro\n' ' InstanceType: t2.micro\n' ' MinCount: 0\n' ' MaxCount: 10\n' ' Networking . . . :\n' ' SubnetIds:\n' ' - subnet-1234567890abcdef0\n', 'cluster_name': 'test-api-cluster'} {'cloud_formation_stack_status': 'CREATE_IN_PROGRESS', 'cloudformation_stack_arn': 'arn:aws:cloudformation:us-east-1:123456789012:stack/test-api-cluster/abcd1234-ef56-gh78-ij90-1234abcd5678', 'cluster_configuration': {'url': 'https://parallelcluster-021345abcdef6789-v1-do-not-delete...}, 'cluster_name': 'test-api-cluster', 'cluster_status': 'CREATE_IN_PROGRESS', 'compute_fleet_status': 'UNKNOWN', 'creation_time': datetime.datetime(2022, 4, 28, 16, 18, 47, 972000, tzinfo=tzlocal()), 'last_updated_time': datetime.datetime(2022, 4, 28, 16, 18, 47, 972000, tzinfo=tzlocal()), 'region': 'us-east-1', 'tags': [{'key': 'parallelcluster:version', 'value': '3.1.3'}], 'version': '3.1.3'} . . . . . . working . . . {'cloud_formation_stack_status': 'CREATE_COMPLETE', 'cloudformation_stack_arn': 'arn:aws:cloudformation:us-east-1:123456789012:stack/test-api-cluster/abcd1234-ef56-gh78-ij90-1234abcd5678', 'cluster_configuration': {'url': 'https://parallelcluster-021345abcdef6789-v1-do-not-delete...}, 'cluster_name': 'test-api-cluster', 'cluster_status': 'CREATE_COMPLETE', 'compute_fleet_status': 'RUNNING', 'creation_time': datetime.datetime(2022, 4, 28, 16, 18, 47, 972000, tzinfo=tzlocal()), 'head_node': {'instance_id': 'i-abcdef01234567890', 'instance_type': 't2.micro', 'launch_time': datetime.datetime(2022, 4, 28, 16, 21, 46, tzinfo=tzlocal()), 'private_ip_address': '172.31.27.153', 'public_ip_address': '52.90.156.51', 'state': 'running'}, 'last_updated_time': datetime.datetime(2022, 4, 28, 16, 18, 47, 972000, tzinfo=tzlocal()), 'region': 'us-east-1', 'tags': [{'key': 'parallelcluster:version', 'value': '3.1.3'}], 'version': '3.1.3'} READY!
-
将以下示例客户端代码复制到
delete_cluster_client.py
。客户端代码发出删除集群的请求。# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: MIT-0 # # Permission is hereby granted, free of charge, to any person obtaining a copy of this # software and associated documentation files (the "Software"), to deal in the Software # without restriction, including without limitation the rights to use, copy, modify, # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # # Author: Evan F. Bollig (Github: bollig) import time, datetime import os import pcluster_client from pprint import pprint from pcluster_client.api import ( cluster_compute_fleet_api, cluster_instances_api, cluster_operations_api ) from pcluster_client.model.create_cluster_request_content import CreateClusterRequestContent from pcluster_client.model.cluster_status import ClusterStatus region=os.environ.get("AWS_DEFAULT_REGION") # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = pcluster_client.Configuration( host = os.environ.get("PCLUSTER_API_URL") ) cluster_name=os.environ.get("CLUSTER_NAME") # Enter a context with an instance of the API client with pcluster_client.ApiClient(configuration) as api_client: cluster_ops = cluster_operations_api.ClusterOperationsApi(api_client) # Delete the cluster gone = False try: api_response = cluster_ops.delete_cluster(cluster_name, region=region) except pcluster_client.ApiException as e: print("Exception when calling delete_cluster: %s\n" % e) time.sleep(60) # Confirm cluster status with describe_cluster while not gone: try: api_response = cluster_ops.describe_cluster(cluster_name, region=region) pprint(api_response) if api_response.cluster_status == ClusterStatus('DELETE_IN_PROGRESS'): print('. . . working . . .', end='', flush=True) time.sleep(60) except pcluster_client.ApiException as e: gone = True print("DELETE COMPLETE or Exception when calling describe_cluster: %s\n" % e)
-
运行以下命令以删除集群。
$
python3 delete_cluster_client.py{'cloud_formation_stack_status': 'DELETE_IN_PROGRESS', 'cloudformation_stack_arn': 'arn:aws:cloudformation:us-east-1:123456789012:stack/test-api-cluster/abcd1234-ef56-gh78-ij90-1234abcd5678', 'cluster_configuration': {'url': 'https://parallelcluster-021345abcdef6789-v1-do-not-delete...}, 'cluster_name': 'test-api-cluster', 'cluster_status': 'DELETE_IN_PROGRESS', 'compute_fleet_status': 'UNKNOWN', 'creation_time': datetime.datetime(2022, 4, 28, 16, 50, 47, 943000, tzinfo=tzlocal()), 'head_node': {'instance_id': 'i-abcdef01234567890', 'instance_type': 't2.micro', 'launch_time': datetime.datetime(2022, 4, 28, 16, 53, 48, tzinfo=tzlocal()), 'private_ip_address': '172.31.17.132', 'public_ip_address': '34.201.100.37', 'state': 'running'}, 'last_updated_time': datetime.datetime(2022, 4, 28, 16, 50, 47, 943000, tzinfo=tzlocal()), 'region': 'us-east-1', 'tags': [{'key': 'parallelcluster:version', 'value': '3.1.3'}], 'version': '3.1.3'} . . . . . . working . . . {'cloud_formation_stack_status': 'DELETE_IN_PROGRESS', 'cloudformation_stack_arn': 'arn:aws:cloudformation:us-east-1:123456789012:stack/test-api-cluster/abcd1234-ef56-gh78-ij90-1234abcd5678', 'cluster_configuration': {'url': 'https://parallelcluster-021345abcdef6789-v1-do-not-delete...}, 'cluster_name': 'test-api-cluster', 'cluster_status': 'DELETE_IN_PROGRESS', 'compute_fleet_status': 'UNKNOWN', 'creation_time': datetime.datetime(2022, 4, 28, 16, 50, 47, 943000, tzinfo=tzlocal()), 'last_updated_time': datetime.datetime(2022, 4, 28, 16, 50, 47, 943000, tzinfo=tzlocal()), 'region': 'us-east-1', 'tags': [{'key': 'parallelcluster:version', 'value': '3.1.3'}], 'version': '3.1.3'} . . . working . . . DELETE COMPLETE or Exception when calling describe_cluster: (404) Reason: Not Found . . . HTTP response body: {"message":"Cluster 'test-api-cluster' does not exist or belongs to an incompatible ParallelCluster major version."}
-
测试完成后,取消设置环境变量。
$
unset AWS_SESSION_TOKEN unset AWS_SECRET_ACCESS_KEY unset AWS_ACCESS_KEY_ID
您可以使用 Amazon Web Services Management Console 或删除 Amazon CLI 您的API.
-
在 Amazon CloudFormation 控制台中,选择API堆栈,然后选择删除。
-
如果使用的是 Amazon CLI,请运行以下命令。
使用 Amazon CloudFormation。
$
aws cloudformation delete-stack --stack-name ${API_STACK_NAME}