Network configurations

Amazon ParallelCluster uses Amazon Virtual Private Cloud (VPC) for networking. VPC provides a flexible and configurable networking platform where you can deploy clusters.

The VPC must have DNS Resolution = yes, DNS Hostnames = yes and DHCP options with the correct domain-name for the Region. The default DHCP Option Set already specifies the required AmazonProvidedDNS. If specifying more than one domain name server, see DHCP options sets in the Amazon VPC User Guide.

Amazon ParallelCluster supports the following high-level configurations:

All of these configurations can operate with or without public IP addressing. Amazon ParallelCluster can also be deployed to use an HTTP proxy for all Amazon requests. The combinations of these configurations result in many deployment scenarios. For example, you can configure a single public subnet with all access over the internet. Or, you can configure a fully private network using Amazon Direct Connect and HTTP proxy for all traffic.

Starting from Amazon ParallelCluster 3.0.0 it is possible to configure different SecurityGroups, AdditionalSecurityGroups and PlacementGroup settings for each queue. For more information, see HeadNode / Networking and SlurmQueues / Networking and AwsBatchQueues / Networking.

For illustrations of some networking scenarios, see the following architecture diagrams.

Amazon ParallelCluster in a single public subnet

The configuration for this architecture requires the following settings:

# Note that all values are only provided as examples
HeadNode:
  ...
  Networking:
    SubnetId: subnet-12345678 # subnet with internet gateway
    #ElasticIp: true | false | eip-12345678
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-12345678 # subnet with internet gateway
        #AssignPublicIp: true

In this configuration, all instances of the cluster must be assigned a public IP in order to get internet access. To achieve this, do the following:

For more information, see Enabling internet access in Amazon VPC User Guide.

Amazon ParallelCluster using two subnets

The configuration to use an existing private subnet for compute instances requires the following settings:

# Note that all values are only provided as examples
HeadNode:
  ...
  Networking:
    SubnetId: subnet-12345678 # subnet with internet gateway
    #ElasticIp: true | false | eip-12345678
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-23456789 # subnet with NAT gateway
        #AssignPublicIp: false

In this configuration, only the head node of the cluster is required to have a public IP assigned. You can achieve this by either turning on the "Enable auto-assign public IPv4 address" setting for the subnet used in HeadNode / Networking / SubnetId or by assigning an Elastic IP in HeadNode / Networking / ElasticIp.

If you define a p4d instance type or another instance type that has multiple network interfaces or a network interface card to the head node, you must set HeadNode / Networking / ElasticIp to true to provide public access. Amazon public IPs can only be assigned to instances launched with a single network interface. For more information on IP addresses, see Assign a public IPv4 address during instance launch in the Amazon EC2 User Guide for Linux Instances.

This configuration requires a NAT gateway or an internal proxy in the subnet used for the queues, to give internet access to the compute instances.

Amazon ParallelCluster in a single private subnet connected using Amazon Direct Connect

The configuration for this architecture requires the following settings:

# Note that all values are only provided as examples
HeadNode:
  ...
  Networking:
    SubnetId: subnet-34567890 # subnet with proxy
    Proxy:
      HttpProxyAddress: http://proxy-address:port
  Ssh:
    KeyName: ec2-key-name
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-34567890 # subnet with proxy
        AssignPublicIp: false
        Proxy:
          HttpProxyAddress: http://proxy-address:port

When Scheduling / SlurmQueues / Networking / AssignPublicIp is set to false, the subnets must be correctly set up to use the Proxy for all traffic. Web access is required for both head and compute nodes.

Amazon ParallelCluster with Amazon Batch scheduler

When you use awsbatch as the scheduler type, Amazon ParallelCluster creates an Amazon Batch managed compute environment. The Amazon Batch environment manages Amazon Elastic Container Service (Amazon ECS) container instances. These instances are launched in the subnet configured in the AwsBatchQueues / Networking / SubnetIds parameter. For Amazon Batch to function correctly, Amazon ECS container instances need external network access to communicate with the Amazon ECS service endpoint. This translates into the following scenarios:

Additionally, if you're interested in multi-node parallel jobs (from the Amazon Batch docs):

Amazon Batch multi-node parallel jobs use the Amazon ECS awsvpc network mode. This gives your multi-node parallel job containers the same networking properties as Amazon EC2 instances. Each multi-node parallel job container gets its own elastic network interface, a primary private IP address, and an internal DNS hostname. The network interface is created in the same Amazon VPC subnet as its host compute resource. Any security groups that are applied to your compute resources are also applied to it.

When using Amazon ECS Task Networking, the awsvpc network mode doesn't provide elastic network interfaces with public IP addresses for tasks that use the Amazon EC2 launch type. To access the internet, tasks that use the Amazon EC2 launch type must be launched in a private subnet that's configured to use a NAT gateway.

You must configure a NAT gateway in order to enable the cluster to run multi-node parallel jobs.

All the previous configuration and considerations are valid for Amazon Batch, too. The following is an example of a Amazon Batch networking configuration.

# Note that all values are only provided as examples
HeadNode:
  ...
  Networking:
    SubnetId: subnet-12345678 # subnet with internet gateway, NAT gateway or proxy
    #ElasticIp: true | false | eip-12345678
    #Proxy:
      #HttpProxyAddress: http://proxy-address:port
  Ssh:
    KeyName: ec2-key-name
Scheduling:
  Scheduler: awsbatch
  AwsBatchQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-23456789 # subnet with internet gateway, NAT gateway or proxy
        #AssignPublicIp: true | false

In the Scheduling / AwsBatchQueues / Networking section, the SubnetIds is a list type but, currently, only one subnet is supported.

For more information, see the following topics:

Amazon ParallelCluster in a single subnet with no internet access

A subnet without internet access doesn't allow inbound or outbound connections to the internet. This Amazon ParallelCluster configuration can help security-concerned customers further enhance the security of their Amazon ParallelCluster resources. Amazon ParallelCluster nodes are built from Amazon ParallelCluster AMIs that include all of the software that's required to run a cluster with no internet access. This way, Amazon ParallelCluster can create and manage clusters with nodes that don't have internet access.

In this section, you learn about how to configure the cluster. You also learn about limitations in running clusters without internet access.

Configuring VPC endpoints

To ensure the proper functioning of the cluster, the cluster nodes must be able to interact with a number of Amazon Services.

Create and configure the following VPC endpoints so that cluster nodes can interact with the Amazon Services, without internet access:

** This endpoint is only required when DirectoryService is enabled, otherwise it is optional.

All instances in the VPC must have proper security groups to communicate with the endpoints. You can do this by adding security groups to AdditionalSecurityGroups under the HeadNode and AdditionalSecurityGroups under the SlurmQueues configurations. For example, if the VPC endpoints are created without explicitly specifying a security group, the default security group is associated with the endpoints. By adding the default security group to AdditionalSecurityGroups, you enable the communication between the cluster and the endpoints.

PolicyDocument:
  Version: 2012-10-17
  Statement:
    - Effect: Allow
      Principal: "*"
      Action:
        - "s3:PutObject"
      Resource:
        - !Sub "arn:${AWS::Partition}:s3:::cloudformation-waitcondition-${AWS::Region}/*"

Disable Route 53 and use Amazon EC2 hostnames

When creating a Slurm cluster, Amazon ParallelCluster creates a private Route 53 hosted zone that is used to resolve the custom compute node hostnames, such as {queue_name}-{st|dy}-{compute_resource}-{N}. Because Route 53 doesn't support VPC endpoints, this feature must be disabled. Additionally, Amazon ParallelCluster must be configured to use the default Amazon EC2 hostnames, such as ip-1-2-3-4. Apply the following settings to your cluster configuration:

...
Scheduling:
  ...
  SlurmSettings:
    Dns:
      DisableManagedDns: true
      UseEc2Hostnames: true

Cluster configuration

Learn how to configure your cluster to run in a subnet with no connection to the internet.

The configuration for this architecture requires the following settings:

# Note that all values are only provided as examples
...
HeadNode:
  ...
  Networking:
    SubnetId: subnet-1234567890abcdef0 # the VPC of the subnet needs to have VPC endpoints
    AdditionalSecurityGroups:
      - sg-abcdef01234567890 # optional, the security group that enables the communication between the cluster and the VPC endpoints
Scheduling:
  Scheduler: Slurm # Cluster in a subnet without internet access is supported only when the scheduler is Slurm.
  SlurmSettings:
    Dns:
      DisableManagedDns: true
      UseEc2Hostnames: true
  SlurmQueues:
    - ...
      Networking:
        SubnetIds:
          - subnet-1234567890abcdef0 # the VPC of the subnet needs to have VPC endpoints attached
        AdditionalSecurityGroups:
          - sg-1abcdef01234567890 # optional, the security group that enables the communication between the cluster and the VPC endpoints

Limitations