View a markdown version of this page

ProductionVariantRoutingConfig - Amazon SageMaker
ContentsSee Also
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

ProductionVariantRoutingConfig

Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

Contents

RoutingStrategy

Sets how the endpoint routes incoming traffic:

  • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

  • RANDOM: The endpoint routes each request to a randomly chosen instance.

  • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

Type: String

Valid Values: LEAST_OUTSTANDING_REQUESTS | RANDOM | PREFIX_AWARE

Required: Yes

PrefixAwareRoutingConfig

The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

Type: PrefixAwareRoutingConfig object

Required: No

See Also

For more information about using this API in one of the language-specific Amazon SDKs, see the following: