Jobs stuck in a RUNNABLE status

If Amazon Batch detects that you have a RUNNABLE job blocking the front of the queue, you'll receive Service job queue blocked events from with the reason. The same reason is also updated into the statusReason field as a part of ListServiceJobs and DescribeServiceJob API calls.

Optionally, you can configure the jobStateTimeLimitActions parameter through CreateJobQueue and UpdateJobQueue API actions. The jobStateTimeLimitActions parameter specifies a set of actions that Amazon Batch performs on jobs in a specific state. You can set a time threshold in seconds through the maxTimeSeconds field.

For job queues connected to SAGEMAKER_TRAINING service environments, the only action you can use with jobStateTimeLimitActions.action is TERMINATE, which will terminate a job.

For example, you can set the jobStateTimeLimitActions parameter to wait up to 4 hours for any job in the RUNNABLE state that is waiting for sufficient capacity to become available. You can do this by setting reason to CAPACITY:INSUFFICIENT_INSTANCE_CAPACITY and maxTimeSeconds to 14400 before terminating the job and allowing the next job to advance to the head of the job queue.

The following sections each describe a distinct type of blocked reason that Amazon Batch can infer. Each section includes: