View a markdown version of this page

Jobs stuck in RUNNABLE due to capacity - Amazon Batch
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Jobs stuck in RUNNABLE due to capacity

The following scenario describes how Amazon Batch identifies insufficient capacity conditions.

Insufficient instance capacity

Amazon Batch interprets two sets of responses received by SageMaker Training jobs as cases of insufficient instance capacity. Whenever these are received, Amazon Batch sets a corresponding statusReason of CAPACITY:INSUFFICIENT_INSTANCE_CAPACITY.

In both cases, the time the corresponding Amazon Batch job spent in SCHEDULED between job submission and failure accumulates between job submissions for comparison against maxTimeSeconds.

Scenario

TrainingJobStatus

Indicator

Training job fails with capacity error

FAILED

failedReason begins with CapacityError

Training job times out waiting for capacity

STOPPED

secondaryStatus indicates the job timed out in pending waiting for capacity

If either of these cases is hit, Amazon Batch will consider the job to be encountering insufficient capacity.

  • statusReason message while the job is stuck: CAPACITY:INSUFFICIENT_INSTANCE_CAPACITY - Unable to provision requested compute resources

  • reason used for jobStateTimeLimitActions: CAPACITY:INSUFFICIENT_INSTANCE_CAPACITY

  • statusReason message after the job is terminated by jobStateTimeLimitActions: Terminated by JobStateTimeLimit action due to reason: CAPACITY:INSUFFICIENT_INSTANCE_CAPACITY