View a markdown version of this page

Deploy models to GPU partitions with inference components - Amazon SageMaker AI
Inference component rulesCreate inference components
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Deploy models to GPU partitions with inference components

After your partitioned endpoint is InService, you deploy a model to it by creating an inference component that requests a partition type. Each copy of the inference component occupies one partition of that type. To create the endpoint, see Create an endpoint with GPU partitions.

Inference component rules

The following rules apply to inference components on an endpoint with a partition layout:

  • AcceleratorPartitionType is required. If you omit it, SageMaker AI returns a 400 error.

  • The requested type must be one that the endpoint layout declares. Otherwise, SageMaker AI returns a 400 error that names the declared types.

  • NumberOfAcceleratorDevicesRequired must be 1. It refers to one partition.

  • MinMemoryRequiredInMb and NumberOfCpuCoresRequired describe host memory and CPU. The partition type sets the GPU memory.

  • Each copy uses one partition. Increase CopyCount to place more copies on free partitions of the same type.

Create inference components

The following procedure deploys a model to the endpoint partition-demo on a mig-4g.20gb partition. The inference component defines its model inline with a Hugging Face inference container that downloads the gpt2 model from Hugging Face at startup, so you don't need to run create-model.

To create inference components on partitions (Amazon CLI)

  1. Create an inference component. Set AcceleratorPartitionType to select the partition type that the model runs on. The following create-inference-component command creates the inference component demo-ic-4g on a mig-4g.20gb partition:

    aws sagemaker create-inference-component \
  --region us-west-2 \
  --inference-component-name demo-ic-4g \
  --endpoint-name partition-demo \
  --variant-name AllTraffic \
  --specification '{
    "Container": {
      "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04",
      "Environment": {
        "HF_MODEL_ID": "gpt2",
        "HF_TASK": "text-generation"
      }
    },
    "ComputeResourceRequirements": {
      "AcceleratorPartitionType": "mig-4g.20gb",
      "NumberOfAcceleratorDevicesRequired": 1,
      "NumberOfCpuCoresRequired": 1,
      "MinMemoryRequiredInMb": 2048
    }
  }' \
  --runtime-config '{"CopyCount": 1}'

    To place a model on each partition type in the layout, repeat the command for each type with a different inference component name and AcceleratorPartitionType.

    The following list describes the fields in this example:

    • AcceleratorPartitionType must be one of the types in the endpoint layout.

    • NumberOfAcceleratorDevicesRequired must be 1. It refers to one partition.

    • NumberOfCpuCoresRequired and MinMemoryRequiredInMb describe host CPU and memory. The partition type sets the GPU memory.

  2. Describe the inference component. It takes a few minutes to create. The following describe-inference-component command returns the name, status, and any failure reason of the inference component:

    aws sagemaker describe-inference-component --region us-west-2 --inference-component-name demo-ic-4g \
  --query '[InferenceComponentName,InferenceComponentStatus,FailureReason]' --output text

    An inference component is ready when its status is InService. If the status is Failed, the third column shows the failure reason. When the inference component is InService, the model runs on its own partition.

Partitioning doesn't affect invocation. Invoke the model as you would any inference component, by calling InvokeEndpoint with the inference component name. For more information, see Invoke models for real-time inference.