InvokeInferenceRouter
Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
Request Syntax
POST /inference-routers/InferenceRouterName/invocations HTTP/1.1
X-Amzn-SageMaker-Routing-Target-Name: RoutingTargetName
Content-Type: ContentType
Accept: Accept
X-Amzn-SageMaker-Inference-Id: InferenceId
X-Amzn-SageMaker-Prefix-Aware-Id: PrefixAwareId
X-Amzn-SageMaker-Custom-Attributes: CustomAttributes
Body
URI Request Parameters
The request uses the following URI parameters.
- Accept
-
The MIME type that you want for the inference response. The router passes it to the destination.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - ContentType
-
The MIME type of the input data in the request body. The router passes it to the destination.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - CustomAttributes
-
Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - InferenceId
-
An identifier for the inference request. The router passes it to the destination.
Length Constraints: Minimum length of 1. Maximum length of 64.
Pattern:
\S[\p{Print}]* - InferenceRouterName
-
The name of the inference router to invoke.
Length Constraints: Minimum length of 1. Maximum length of 63.
Pattern:
[a-zA-Z0-9](-*[a-zA-Z0-9])*Required: Yes
- PrefixAwareId
-
An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.
Applies only to destination endpoints configured with a
RoutingStrategyofPREFIX_AWARE.Length Constraints: Minimum length of 1. Maximum length of 64.
Pattern:
\S[\p{Print}]* - RoutingTargetName
-
The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.
Length Constraints: Minimum length of 1. Maximum length of 63.
Pattern:
[a-zA-Z0-9](-*[a-zA-Z0-9])*Required: Yes
Request Body
The request accepts the following binary data.
- Body
-
The input data for the inference, in the format that your model expects. The body can't be empty.
Length Constraints: Minimum length of 0. Maximum length of 6291456.
Required: Yes
Response Syntax
HTTP/1.1 200
Content-Type: ContentType
X-Amzn-SageMaker-Invoked-Routed-Resource-Arn: InvokedRoutedResourceArn
X-Amzn-SageMaker-Invoked-Routing-Reason: RoutingReason
X-Amzn-Invoked-Production-Variant: InvokedProductionVariant
X-Amzn-SageMaker-Custom-Attributes: CustomAttributes
Body
Response Elements
If the action is successful, the service sends back an HTTP 200 response.
The response returns the following HTTP headers.
- ContentType
-
The MIME type of the inference returned in the response body.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - CustomAttributes
-
Provides additional information in the response about the inference. This is the value that the model container of the destination returned, unchanged.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - InvokedProductionVariant
-
The production variant of the destination endpoint that served the request.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - InvokedRoutedResourceArn
-
The Amazon Resource Name (ARN) of the endpoint or inference component that served the request.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* - RoutingReason
-
The reason that the router chose the destination that served the request. Possible values:
-
local-preferred– The destination is a healthy destination in the router's Region. -
capacity-available– No destination in the router's Region could take the request, and two or more destinations in other Regions could. -
only-healthy– No destination in the router's Region could take the request, and exactly one destination in another Region could. -
failover– The router moved the request away from destinations in its Region because they were overloaded or returning errors. You also see this value when the router can't assess destination health, and when no destination is healthy and the router tries one anyway.
Length Constraints: Minimum length of 0. Maximum length of 1024.
Pattern:
\p{ASCII}* -
The response returns the following as the HTTP body.
- Body
-
The inference that the model returned.
Length Constraints: Minimum length of 0. Maximum length of 6291456.
Errors
For information about the errors that are common to all actions, see Common Error Types.
- AccessDeniedException
-
You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
- Message
-
A message that describes the error.
HTTP Status Code: 403
- InternalDependencyException
-
The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
- Message
-
A message that describes the error.
HTTP Status Code: 530
- InternalFailure
-
An internal error occurred. Retry the request with exponential backoff. The Amazon SDKs and the Amazon CLI do this by default.
- Message
-
A message that describes the error.
HTTP Status Code: 500
- ModelError
-
The model in the destination that served the request returned an error. Check
OriginalStatusCode,OriginalMessage, and the destination's logs in Amazon CloudWatch.- LogStreamArn
-
The Amazon Resource Name (ARN) of the Amazon CloudWatch log stream for the destination that served the request.
- Message
-
A message that describes the error.
- OriginalMessage
-
The error message that the model container returned.
- OriginalStatusCode
-
The HTTP status code that the model container returned.
HTTP Status Code: 424
- ModelNotReadyException
-
The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
- Message
-
A message that describes the error.
HTTP Status Code: 429
- RequestTimeoutException
-
The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
- Message
-
A message that describes the error.
HTTP Status Code: 408
- ServiceUnavailableException
-
The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon SDKs and the Amazon CLI do this by default.
- Message
-
A message that describes the error.
HTTP Status Code: 503
- ThrottlingException
-
The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
- Message
-
A message that describes the error.
HTTP Status Code: 429
- ValidationException
-
The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
- Message
-
A message that describes the error.
HTTP Status Code: 400
See Also
For more information about using this API in one of the language-specific Amazon SDKs, see the following: