Working with Spot Instances
Amazon ParallelCluster uses Spot Instances if you have set SlurmQueues /
CapacityType or AwsBatchQueues / CapacityType to SPOT
in the cluster configuration file. Spot Instances are more cost effective than On-Demand Instances, but they might be interrupted. It might help to
take advantage of Spot Instance interruption notices, which provide a two-minute warning before Amazon EC2 must stop or terminate
your Spot Instance. For more information, see Spot Instance interruptions in
Amazon EC2 User Guide for Linux Instances. To learn how AwsBatchQueues works with Spot Instances, see Compute Resources
in the Amazon Batch User Guide.
The Amazon ParallelCluster configured scheduler assigns jobs to compute resources in queues with spot instances in the same way it assigns jobs to compute resources in queues with on-demand instances.
When using Spot Instances, an AWSServiceRoleForEC2Spot service-linked role must exist in your account. To create this role in your account using the Amazon CLI, run the following command:
$aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
For more information, see Service-linked role for Spot Instance requests in the Amazon EC2 User Guide for Linux Instances.
The following sections describe three scenarios in which Spot Instances can be interrupted when using SlurmQueues.
Scenario 1: Spot Instance with no running jobs is interrupted
When this interruption occurs, Amazon ParallelCluster tries to replace the instance if the scheduler queue has pending jobs that require additional instances, or if the number of active instances is lower than the SlurmQueues / ComputeResources / MinCount. If Amazon ParallelCluster can't provision new instances, then a request for new instances is periodically repeated.
Scenario 2: Spot Instance running single node jobs is interrupted
The job fails with a state code of NODE_FAIL, and the job is requeued (unless
--no-requeue is specified when the job is submitted). If the node is a static node, it's replaced.
If the node is a dynamic node, the node is terminated and reset. For more information about sbatch,
including the --no-requeue parameter, see sbatch
Scenario 3: Spot Instance running multi-node jobs is interrupted
The job fails with a state code of NODE_FAIL, and the job is requeued (unless
--no-requeue was specified when the job was submitted). If the node is a static node, it's replaced.
If the node is a dynamic node, the node is terminated and reset. Other nodes that were running the terminated jobs
might be allocated to other pending jobs, or scaled down after the configured SlurmSettings /
ScaledownIdletime time has passed.
For more information about Spot Instances, see Spot Instances in the Amazon EC2 User Guide for Linux Instances.