Release notes and document history

Documentation only release:

• Amazon ParallelCluster version 2 has its own separate user guide.

Bug fixes:

• Prevent replacement of managed FSx for Lustre file systems and loss of data on cluster updates that include changes to vpc_security_group_id.

Changes:

• Upgrade Intel MPI Library to Version 2021 Update 6 (updated from Version 2021 Update 4). For more information, see Intel® MPI Library 2021 Update 6.

• Upgrade EFA installer to 1.19.0

  • Efa-driver: efa-1.16.0-1

  • Efa-config: efa-config-1.11-1 (from efa-config-1.9-1)

  • Efa-profile: efa-profile-1.5-1 (no change)

  • Libfabric-aws: libfabric-aws-1.16.0-1 (from libfabric-1.13.2)

  • Rdma-core: rdma-core-41.0-2 (from rdma-core-37.0)

  • Open MPI: openmpi40-aws-4.1.4-3 (from openmpi40-aws-4.1.1-2)

• Upgrade Python runtime, that's used by Lambda functions in the Amazon Batch integration, to python3.9.

Bug fixes:

• Prevent cluster tags from being changed during an update because it's not supported.

Changes:

• Upgrade Slurm to version 20.11.9.

Enhancements:

• Improve exception management in case of missing networking.

Changes:

• OS package updates and security fixes.

Enhancements:

• Add support for NEW_CHANGED_DELETED as value of FSx for Lustre AutoImportPolicy option.

• Remove support for SGE and Torque schedulers.

• Disable log4j-cve-2021-44228-hotpatch service on Amazon Linux to avoid incurring potential performance degradation.

Changes:

• Upgrade NVIDIA driver to version 470.103.01 (from 470.82.01).

• Upgrade NVIDIA Fabric manager to version 470.103.01 (from 470.82.01).

• Upgrade CUDA library to version 11.4.4 (from 11.4.3).

• Intel MPI updated to Version 2021 Update 4 (updated from Version 2019 Update 8). For more information, see Intel® MPI Library 2021 Update 4.

• Extend head node creation timeout to one hour.

Bug fixes:

• Fix DCV connection through browsers.

• Fix YAML quoting to prevent custom Tags from being parsed as numbers.

• CentOS 8 support removed. CentOS 8 reaches end of life (EOL) on December 31, 2021.

• Upgrade Slurm Workload Manager to version 20.11.8.

• Upgrade Cinc Client to 17.2.29.

• NICE DCV updated to NICE DCV 2021.2-11190. For more information, see DCV 2021.2-11190— October 11, 2021 in the NICE DCV Administrator Guide.

• Upgrade NVIDIA driver to version 470.82.01 (from 460.73.01).

• Upgrade CUDA library to version 11.4.3 (from 11.3.0).

• Upgrade NVIDIA Fabric Manager to 470.82.01.

• Disable package update at instance launch time on Amazon Linux 2.

• Disable unattended package update on Ubuntu and Amazon Linux 2.

• Install Python 3 version of Amazon CloudFormation helper scripts on CentOS 7 and Ubuntu 18.04. (These were already used on Amazon Linux 2 and Ubuntu 20.04.)

• Disable update of ec2_iam_role parameter.

• Fix the CpuOptions configuration in the launch template for T2 instances.

• Fix pcluster createami failure due to Son of Grid Engine sources not being available at arc.liv.ac.uk.

Upgrade Elastic Fabric Adapter installer to 1.14.1 (from 1.13.0)

• EFA config: efa-config-1.9-1 (from efa-config-1.9)

• EFA profile: efa-profile-1.5-1 (no change)

• EFA Kernel module: efa-1.14.2 (from efa-1.13.0)

• RDMA core: rdma-core-37.0 (from rdma-core-35.0amzn)

• Libfabric: libfabric-1.13.2 (from libfabric-1.13.0amzn1.0)

• Open MPI: openmpi40-aws-4.1.1-2 (no change)

GPUDirect RDMA is always enabled if supported by the instance type.

• The enable_efa_gdr and enable_efa_gdr configuration options have no effect.

• Do not install EFA with GPUDirect RDMA (GDR) enabled at bootstrap time if EFA is installed in the base AMI.

• Lock version of nvidia-fabricmanager package to remain in sync with the NVIDIA driver version installed by Amazon ParallelCluster.

• Slurm: Fix issue caused when cluster was stopped and restarted while a node was powering up.

• Elastic Fabric Adapter installer updated to 1.13.0:

  • EFA config: efa-config-1.9 (no change)

  • EFA profile: efa-profile-1.5-1 (no change)

  • EFA Kernel module: efa-1.13.0 (no change)

  • RDMA core: rdma-core-35.0amzn (from rdma-core-32.1amzn)

  • Libfabric: libfabric-1.13.0amzn1.0 (from libfabric-1.11.2amzn1.1)

  • Open MPI: openmpi40-aws-4.1.1-2 (no change)

• When using a custom AMI with a preinstalled EFA package, no changes are made to EFA at node bootstrap time. The original EFA package deployment is preserved.

• Mount file systems using the noatime mount option to stop recording last access time when a file is read. This improves the performance of the remote file system.

• Elastic Fabric Adapter installer updated to 1.12.3:

  • EFA config: efa-config-1.9 (from efa-config-1.8-1)

  • EFA profile: efa-profile-1.5-1 (no change)

  • EFA Kernel module: efa-1.13.0 (from efa-1.12.3)

  • RDMA core: rdma-core-32.1amzn (no change)

  • Libfabric: libfabric-1.11.2amzn1.1 (no change)

  • Open MPI: openmpi40-aws-4.1.1-2 (no change)

• Retry installations of the aws-parallelcluster package on the head node when using Amazon Batch as the scheduler.

• Avoid failures when building SGE on an instance type with more than 31 vCPUs.

• Pinned to version 1.247347.6 of the Amazon CloudWatch Agent to avoid issues seen in version 1.247348.0.

• Added support for Ubuntu 20.04 (ubuntu2004) and removed support for Ubuntu 16.04 (ubuntu1604) and Amazon Linux (alinux). Amazon Linux 2 (alinux2) remains fully supported. For more information, see base_os.

• Removed support for Python versions below 3.6.

• Default root volume size increased to 35 gibibytes (GiB). For more information, see compute_root_volume_size and master_root_volume_size.

• Elastic Fabric Adapter installer updated to 1.12.2:

  • EFA config: efa-config-1.8-1 (from efa-config-1.7)

  • EFA profile: efa-profile-1.5-1 (from efa-profile-1.4)

  • EFA Kernel module: efa-1.12.3 (from efa-1.10.2)

  • RDMA core: rdma-core-32.1amzn (from rdma-core-31.2amzn)

  • Libfabric: libfabric-1.11.2amzn1.1 (from libfabric-1.11.1amzn1.0)

  • Open MPI: openmpi40-aws-4.1.1-2 (from openmpi40-aws-4.1.0)

• Upgraded Slurm to version 20.11.7 (from 20.02.7).

• Install SSM Agent on centos7 and centos8. (SSM Agent is preinstalled in alinux2, ubuntu1804, and ubuntu2004.)

• SGE: Always use the shortname as the hostname filter with qstat.

• Use instance metadata service Version 2 (IMDSv2) instead of instance metadata service Version 1 (IMDSv1) to retrieve instance metadata. For more information, see Instance metadata and user data in the Amazon EC2 User Guide.

• Upgrade NVIDIA driver to version 460.73.01 (from 450.80.02).

• Upgrade CUDA library to version 11.3.0 (from 11.0).

• Upgrade NVIDIA Fabric Manager to nvidia-fabricmanager-460.

• Upgrade Python used in Amazon ParallelCluster virtualenvs to 3.7.10 (from 3.6.13).

• Upgrade Cinc Client to 16.13.16.

• Upgrade third-party dependencies of aws-parallelcluster-cookbook:

  • apt-7.4.0 (from apt-7.3.0).

  • iptables-8.0.0 (from iptables-7.1.0).

  • line-4.0.1 (from line-2.9.0).

  • openssh-2.9.1 (from openssh-2.8.1).

  • pyenv-3.4.2 (from pyenv-3.1.1).

  • selinux-3.1.1 (from selinux-2.1.1).

  • ulimit-1.1.1 (from ulimit-1.0.0).

  • yum-6.1.1 (from yum-5.1.0).

  • yum-epel-4.1.2 (from yum-epel-3.3.0).

• Upgraded Slurm to version 20.02.7 (from 20.02.4).

• Added support for Ubuntu 18.04 and Amazon Linux 2 on Arm-based Amazon Graviton instances in the Amazon China and Amazon GovCloud (US) Amazon Web Services Regions.

• Elastic Fabric Adapter installer updated to 1.11.2:

  • EFA config: efa-config-1.7 (no change)

  • EFA profile: efa-profile-1.4 (from efa-profile-1.3)

  • EFA Kernel module: efa-1.10.2 (no change)

  • RDMA core: rdma-core-31.2amzn (no change)

  • Libfabric: libfabric-1.11.1amzn1.0 (no change)

  • Open MPI: openmpi40-aws-4.1.0 (no change)

• Improve cluster config validation to use the cluster target AMI when invoking the Amazon EC2 RunInstances API operation in --dry-run mode.

• Update the Python version used in the Amazon ParallelCluster virtual environments to 3.6.13.

• Fix sanity_check for Arm instance types.

• Fix enable_efa when using centos8 with the Slurm scheduler or Arm instance types.

• Run apt update in non-interactive mode (-y).

• Fix encrypted_ephemeral = true with alinux2 and centos8.

• Added support for the Africa (Cape Town) (af-south-1), Europe (Milan) (me-south-1), and Middle East (Bahrain) (me-south-1) Amazon Web Services Regions. At launch, support is limited in the following ways:

  • FSx for Lustre and Arm-based Graviton instances aren't supported in any of these Amazon Web Services Regions.

  • Amazon Batch isn't supported in Africa (Cape Town).

  • Amazon EBS io2 and gp3 volume types aren't supported in the Africa (Cape Town) and Europe (Milan) Amazon Web Services Regions.

• Added support for the Amazon EBS io2 and gp3 volume types. For more information, see [ebs] section and [raid] section.

• Added support for Elastic Fabric Adapter on Arm-based Graviton2 instances running alinux2, ubuntu1804, or ubuntu2004. For more information, see Elastic Fabric Adapter.

• Install the Arm Performance Libraries 20.2.1 on Arm AMIs (alinux2, centos8, and ubuntu1804). For more information, see Arm Performance Libraries.

• Intel MPI updated to Version 2019 Update 8 (updated from Version 2019 Update 7). For more information, see Intel® MPI Library 2019 Update 8.

• Removed the Amazon CloudFormation DescribeStacks API operation call from the Amazon Batch Docker entrypoint to end the job failures caused by throttling by Amazon CloudFormation.

• Improved the calls to Amazon EC2 DescribeInstanceTypes API operation call when validating a cluster configuration.

• Amazon Linux 2 Docker images are pulled from Amazon ECR Public when building the Docker image for the awsbatch scheduler.

• The default instance type changed from the hardcoded t2.micro instance type to the Free Tier instance type for the Amazon Web Services Region (t2.micro or t3.micro, depending on the Amazon Web Services Region). Amazon Web Services Regions that do not have a Free Tier default to the t3.micro instance type.

• Elastic Fabric Adapter installer updated to 1.11.1:

  • EFA config: efa-config-1.7 (from efa-config-1.5)

  • EFA profile: efa-profile-1.3 (from efa-profile-1.1)

  • EFA Kernel module: efa-1.10.2 (no change)

  • RDMA core: rdma-core-31.2amzn (from rdma-core-31.amzn0)

  • Libfabric: libfabric-1.11.1amzn1.0 (from libfabric-1.10.1amzn1.1)

  • Open MPI: openmpi40-aws-4.1.0 (from openmpi40-aws-4.0.5)

• The vpc_settings, vpc_id, and master_subnet_id parameters are now required.

• The nfsd daemon in the head node is now set to use at least 8 threads. If there are more than 8 cores, it will use as many threads as there are cores. When ubuntu1604 is used, the setting only changes after the node is rebooted.

• NICE DCV updated to NICE DCV 2020.2-9662. For more information, see DCV 2020.2-9662— December 04, 2020 in the NICE DCV Administrator Guide.

• The Intel MPI and HPC packages for Amazon ParallelCluster are pulled from Amazon S3. They are no longer pulled from Intel yum repos.

• Changed the default systemd runlevel to multi-user.target on all OSs during creation of official Amazon ParallelCluster AMIs. The runlevel is set to graphical.target on the head node only when DCV is enabled. This prevents graphical services (such as x/gdm) from running when they are not required.

• Enabled support for p4d.24xlarge instances on the head node.

• Increase the maximum number of retry attempts when registering Slurm nodes in Amazon Route 53.

• Added support for CentOS 8 in all Amazon Web Services Regions (outside of the Amazon China and Amazon GovCloud (US) Regions). Removed support for CentOS 6.

• Added support for p4d.24xlarge instances for compute nodes.

• Added support for NVIDIA GPUDirect RDMA on EFA by using the new enable_efa_gdr setting.

• Added support for Amazon FSx for Lustre features.

  • Configure your Amazon FSx for Lustre file system to import preferences using the auto_import_policy setting.

  • Added support for HDD-based Amazon FSx for Lustre file systems using the storage_type and drive_cache_type settings.

• Added a Amazon CloudWatch dashboard, including head node metrics and easy access to cluster logs. For more information, see Amazon CloudWatch dashboard.

• Added support for using an existing Amazon S3 bucket to store cluster configuration information, using the cluster_resource_bucket setting.

• Enhanced the pcluster createami command.

  • Added --post-install parameter to use a post-installation script when build an AMI.

  • Added a validation step to fail when using a base AMI created by a different version of Amazon ParallelCluster.

  • Added a validation step to fail when if the select OS is different from the OS in the base AMI.

  • Added support for using a Amazon ParallelCluster base AMI.

• Enhanced the pcluster update command.

  • The tags setting can now be changed during an update.

  • Queues can now be resized during an update without stopping the compute fleet

• Added all_or_nothing_batch configuration parameter for slurm_resume script. When True, slurm_resume will succeed only if all the instances required by all the pending jobs in Slurm will be available. For more information, see Introducing all_or_nothing_batch launches in the Amazon ParallelCluster Wiki on GitHub.

• Elastic Fabric Adapter installer updated to 1.10.1:

  • EFA config: efa-config-1.5 (from efa-config-1.4)

  • EFA profile: efa-profile-1.1 (from efa-profile-1.0.0)

  • EFA Kernel module: efa-1.10.2 (from efa-1.6.0)

  • RDMA core: rdma-core-31.amzn0 (from rdma-core-28.amzn0)

  • Libfabric: libfabric-1.11.1amzn1.0 (from libfabric-1.10.1amzn1.1)

  • Open MPI: openmpi40-aws-4.0.5 (from openmpi40-aws-4.0.3)

• In the Amazon GovCloud (US) Regions, enable support for NICE DCV and Amazon Batch.

• In the Amazon China Regions, enable support for Amazon FSx for Lustre.

• Upgrade NVIDIA driver to version 450.80.02 (from 450.51.05).

• Install NVIDIA Fabric Manager to enable NVIDIA NVSwitch on supported platforms.

• Removed default Amazon Web Services Region of us-east-1. The default uses this lookup order.

  • Amazon Web Services Region specified in -r or --region argument.

  • AWS_DEFAULT_REGION environment variable.

  • aws_region_name setting in the [aws] section of the Amazon ParallelCluster configuration file (defaults to ~/.parallelcluster/config).

  • region setting in the [default] section of the Amazon CLI configuration file (defaults to ~/aws/config).

• Added support for multiple queues and multiple instance types in the compute fleet when used with Slurm Workload Manager. When using queues, Auto Scaling groups are no longer used on Slurm. An Amazon Route 53 hosted zone is now created with the cluster and is used for DNS resolution of compute nodes when the Slurm scheduler is used. For more information, see Multiple queue mode.

• Added support for NICE DCV on Arm-based Amazon Graviton-based instances.

• Added support for disabling hyperthreading on instance types that do not support CPU options in launch templates (for example *.metal instance types).

• Added support for NFS 4 for file systems shared from the head node.

• Removed dependency on cfn-init when bootstrapping compute nodes to avoid throttling by Amazon CloudFormation when a large number of nodes join the cluster.

• Elastic Fabric Adapter installer updated to 1.9.5:

  • EFA config: efa-config-1.4 (from efa-config-1.3)

  • EFA profile: efa-profile-1.0.0 (new)

  • Kernel module: efa-1.6.0 (no change)

  • RDMA core: rdma-core-28.amzn0 (no change)

  • Libfabric: libfabric-1.10.1amzn1.1 (no change)

  • Open MPI: openmpi40-aws-4.0.3 (no change)

• Upgraded Slurm to version 20.02.4 (from 19.05.5).

• NICE DCV updated to NICE DCV 2020.1-9012. For more information, see DCV 2020.1-9012— August 24, 2020 Release Notes in the NICE DCV Administrator Guide.

• When mounting shared NFS drives, use the head node private IP address instead of the hostname.

• Added new log streams to CloudWatch Logs: chef-client, clustermgtd, computemgtd, slurm_resume, and slurm_suspend.

• Added support for queue names in pre-install and post-install scripts.

• In the Amazon GovCloud (US) Amazon Web Services Regions, use the Amazon DynamoDB on-demand billing option. For more information, see On-Demand Mode in the Amazon DynamoDB Developer Guide.

• Disable screen lock for NICE DCV sessions to prevent users from being locked out.

• Fix pcluster configure when including an Arm-based Amazon Graviton-based instance type.

• Added support for Arm-based Amazon Graviton-based instances (like the A1 and C6g).

• Added support for the automatic daily backup features of Amazon FSx for Lustre. For more information, see automatic_backup_retention_days, copy_tags_to_backups, daily_automatic_backup_start_time, and fsx_backup_id.

• Removed dependency on Berkshelf from pcluster createami.

• Improved the robustness and user experience of pcluster update. For more information, see Using pcluster update.

• Elastic Fabric Adapter installer updated to 1.9.4:

  • Kernel module: efa-1.6.0 (updated from efa-1.5.1)

  • RDMA core: rdma-core-28.amzn0 (updated from rdma-core-25.0)

  • Libfabric: libfabric-1.10.1amzn1.1 (updated from libfabric-aws-1.9.0amzn1.1)

  • Open MPI: openmpi40-aws-4.0.3 (no change)

• Upgrade NVIDIA driver to Tesla version 440.95.01 on CentOS 6 and version 450.51.05 on all other distributions.

• Upgrade CUDA library to version 11.0 on all distributions other than CentOS 6.

• base_os is now a required parameter.

• scheduler is now a required parameter.

• NICE DCV updated to NICE DCV 2020.0. For more information, see NICE DCV releases version 2020.0 with surround sound 7.1 and stylus support.

  Intel MPI updated to Version 2019 Update 7 (updated from Version 2019 Update 6). For more information, see Intel® MPI Library 2019 Update 7.

  Elastic Fabric Adapter installer updated to 1.8.4:

  • Kernel module: efa-1.5.1 (no change)

  • RDMA core: rdma-core-25.0 (no change)

  • Libfabric: libfabric-aws-1.9.0amzn1.1 (no change)

  • Open MPI: openmpi40-aws-4.0.3 (updated from openmpi40-aws-4.0.2)

• Upgrade CentOS 7 AMI to version 7.8-2003 (updated from 7.7-1908). For more information, see CentOS-7 (2003) Release Notes.

• Removed cfn-init-cmd and cfn-wire from logs stored in Amazon CloudWatch Logs. For more information, see Integration with Amazon CloudWatch Logs.

• Added support for Amazon Linux 2.

• Now Amazon CloudWatch Logs is used to collect cluster and scheduler logs. For more information, see Integration with Amazon CloudWatch Logs.

• Added support for new Amazon FSx for Lustre deployment types SCRATCH_2 and PERSISTENT_1. Support for FSx for Lustre on Ubuntu 18.04 and Ubuntu 16.04. For more information, see fsx.

• Added support for NICE DCV on Ubuntu 18.04. For more information, see Connect to the head node through NICE DCV.