Using the Amazon CLI commands for the SageMaker HyperPod APIs

First, prepare and upload lifecycle scripts to an S3 bucket. During cluster creation, HyperPod runs them in each instance group. Upload lifecycle scripts to S3 using the following command.

aws s3 sync \
    ~/local-dir-to-lifecycle-scripts/* \ 
    s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src

If you are starting from scratch, use sample lifecycle scripts provided in the Awsome Distributed Training GitHub repository. The following sub-steps show how to download, what to modify, and how to upload the sample lifecycle scripts to an S3 bucket.

1. Download a copy of the lifecycle script samples to a directory on your local computer.

   git clone https://github.com/aws-samples/awsome-distributed-training/

2. Go into the directory 1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config, where you can find a set of lifecycle scripts.

   cd awsome-distributed-training/1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config

   To learn more about the lifecycle script samples, see Prepare lifecycle scripts for setting up Slurm on SageMaker HyperPod.

3. Write a Slurm configuration file and save it as provisioning_params.json. In the file, specify basic Slurm configuration parameters to properly assign Slurm nodes to the SageMaker HyperPod cluster instance groups. In this tutorial, set up three Slurm nodes named my-controller-group, my-login-group, and worker-group-1, as shown in the following example configuration provisioning_params.json.

   {
       "version": "1.0.0",
       "workload_manager": "slurm",
       "controller_group": "my-controller-group",
       "login_group": "my-login-group",
       "worker_groups": [
           {
               "instance_group_name": "worker-group-1",
               "partition_name": "partition-1"
           }
       ]
   }

4. Upload the scripts to s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src. You can do so by using the S3 console, or by running the following Amazon CLI S3 command.

   aws s3 sync \
       ~/local-dir-to-lifecycle-scripts/* \ 
       s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src

Prepare a CreateCluster request file in JSON format and save as create_cluster.json. The following request template aligns with the Slurm node configuration defined in the provisioning_params.json in Step 1.c. For ExecutionRole, provide the ARN of the IAM role you created with the managed AmazonSageMakerClusterInstanceRolePolicy in SageMaker HyperPod prerequisites.

{
    // Required: Specify the name of the cluster.
    "ClusterName": "my-hyperpod-cluster",
    // Required: Configure instance groups to be launched in the cluster
    "InstanceGroups": [
        {
            // Required: Specify the basic configurations to set up a controller node.
            "InstanceGroupName": "my-controller-group",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "LifeCycleConfig": {
              "SourceS3Uri": "s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src",
              "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "${ROLE}",
            // Optional: Configure an additional storage per instance group.
            "InstanceStorageConfigs": [
                {
                   // Attach an additional EBS volume to each instance within the instance group.
                   // The default mount path for the additional EBS volume is /opt/sagemaker.
                   "EbsVolumeConfig":{
                      // Specify an integer between 1 and 16384 in gigabytes (GB).
                      "VolumeSizeInGB": integer,
                   }
                }
            ]
        },
        {
            "InstanceGroupName": "my-login-group",
            "InstanceType": "ml.m5.4xlarge",
            "InstanceCount": 1,
            "LifeCycleConfig": {
              "SourceS3Uri": "s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src",
              "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "${ROLE}"
        },
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.trn1.32xlarge",
            "InstanceCount": 1,
            "LifeCycleConfig": {
              "SourceS3Uri": "s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src",
              "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "${ROLE}"
        }
    ]
}

Run the following command to create the cluster.

aws sagemaker create-cluster --cli-input-json file://complete/path/to/create_cluster.json

This should return the ARN of the created cluster.

If you receive an error due to resource limits, ensure that you change the instance type to one with sufficient quotas in your account, or request additional quotas by following at SageMaker HyperPod quotas.

Run describe-cluster to check the status of the cluster.

aws sagemaker describe-cluster --cluster-name my-hyperpod-cluster

After the status of the cluster turns to InService, proceed to the next step.

Run list-cluster-nodes to check the details of the cluster nodes.

aws sagemaker list-cluster-nodes --cluster-name my-hyperpod-cluster

This returns a response, and the InstanceId is what your cluster users need for logging (aws ssm) into them. For more information about logging into the cluster nodes and running ML workloads, see Run jobs on SageMaker HyperPod clusters.