View a markdown version of this page

Configure model auto scaling with the console - Amazon SageMaker AI
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Configure model auto scaling with the console

Note

You configure auto scaling differently depending on the endpoint type. For a Single Model Endpoint (SME), follow the next procedure, which scales the number of instances for the production variant. For an endpoint that hosts Inference Components, follow the Inference Component procedure later in this section, which scales the number of copies of each Inference Component.

To configure auto scaling for a model (console)

  1. Open the Amazon SageMaker AI console at https://console.amazonaws.cn/sagemaker/.

  2. On the navigation pane, choose Inference, and then choose Endpoints.

  3. Choose your endpoint, and then for Endpoint runtime settings, choose the variant.

  4. Choose Configure auto scaling.

  5. On the Configure variant automatic scaling page, for Variant automatic scaling, do the following:

    1. For Minimum instance count, type the minimum number of instances that you want the scaling policy to maintain. At least 1 instance is required.

    2. For Maximum instance count, type the maximum number of instances that you want the scaling policy to maintain.

  6. For Built-in scaling policy, do the following:

    1. For the Target metric, SageMakerVariantInvocationsPerInstance is automatically selected for the metric and cannot be changed.

    2. For the Target value, type the average number of invocations per instance per minute for the model. To determine this value, follow the guidelines in Load testing.

    3. (Optional) For Scale-in cool down (seconds) and Scale-out cool down (seconds), enter the amount of time, in seconds, for each cool down period.

    4. (Optional) Select Disable scale in if you don’t want auto scaling to terminate instances when traffic decreases.

  7. Choose Save.

This procedure registers a model as a scalable target with Application Auto Scaling. When you register a model, Application Auto Scaling performs validation checks to ensure the following:

  • The model exists

  • The permissions are sufficient

  • You aren't registering a variant with an instance that is a burstable performance instance such as T2

    Note

    SageMaker AI doesn't support auto scaling for burstable instances such as T2, because they already allow for increased capacity under increased workloads. For information about burstable performance instances, see Amazon EC2 instance types.

For an endpoint that hosts Inference Components, you configure auto scaling for each Inference Component on the Inference components tab. Auto scaling adjusts the number of copies of the Inference Component in response to traffic.

To configure auto scaling for an Inference Component (console)

  1. Open the Amazon SageMaker AI console at https://console.amazonaws.cn/sagemaker/.

  2. In the navigation pane, choose Inference, and then choose Endpoints.

  3. Choose the endpoint.

  4. Choose the Inference components tab.

  5. Select the Inference Component, and then choose Actions.

  6. Choose Configure auto scaling.

  7. Set the minimum and maximum number of copies and the target value, and then choose Save.