Amazon ParallelCluster with Amazon Batch scheduler - Amazon ParallelCluster
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon ParallelCluster with Amazon Batch scheduler

When you use awsbatch as the scheduler type, Amazon ParallelCluster creates an Amazon Batch managed compute environment. The Amazon Batch environment manages Amazon Elastic Container Service (Amazon ECS) container instances. These instances are launched in the subnet configured in the AwsBatchQueues / Networking / SubnetIds parameter. For Amazon Batch to function correctly, Amazon ECS container instances need external network access to communicate with the Amazon ECS service endpoint. This translates into the following scenarios:

  • The Subnet ID specified for the queue uses a NAT gateway to access the internet. We recommended this approach.

  • Instances launched in the queue subnet have public IP addresses and can reach the internet through an Internet Gateway.

Additionally, if you're interested in multi-node parallel jobs (from the Amazon Batch docs):

Amazon Batch multi-node parallel jobs use the Amazon ECS awsvpc network mode. This gives your multi-node parallel job containers the same networking properties as Amazon EC2 instances. Each multi-node parallel job container gets its own elastic network interface, a primary private IP address, and an internal DNS hostname. The network interface is created in the same Amazon VPC subnet as its host compute resource. Any security groups that are applied to your compute resources are also applied to it.

When using Amazon ECS Task Networking, the awsvpc network mode doesn't provide elastic network interfaces with public IP addresses for tasks that use the Amazon EC2 launch type. To access the internet, tasks that use the Amazon EC2 launch type must be launched in a private subnet that's configured to use a NAT gateway.

You must configure a NAT gateway in order to enable the cluster to run multi-node parallel jobs.

ParallelCluster with a NAT Gateway

All the previous configuration and considerations are valid for Amazon Batch, too. The following is an example of a Amazon Batch networking configuration.

# Note that all values are only provided as examples
HeadNode:
  ...
  Networking:
    SubnetId: subnet-12345678 # subnet with internet gateway, NAT gateway or proxy
    #ElasticIp: true | false | eip-12345678
    #Proxy:
      #HttpProxyAddress: http://proxy-address:port
  Ssh:
    KeyName: ec2-key-name
Scheduling:
  Scheduler: awsbatch
  AwsBatchQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-23456789 # subnet with internet gateway, NAT gateway or proxy
        #AssignPublicIp: true | false

In the Scheduling / AwsBatchQueues / Networking section, the SubnetIds is a list type but, currently, only one subnet is supported.

For more information, see the following topics: