Resolving OS configuration changes that cause errors or failures

Check if you can switch to the pcluster-admin user:

$ su - pcluster-admin

If you see an error like cannot change locale...no such file or directory, this confirms the issue.

Check available locales:

$ localedef --list

If this returns an empty list or doesn't contain the default locale, your locale configuration is broken.

Check the last yum command:

$ yum history
$ yum history info #ID

If the last ID doesn't have Return-Code: Success, the post-install scripts might not have run successfully.

Check the chef-client log for package-related errors:

$ less /var/log/chef-client.log

Look for package dependency conflicts in the cfn-init log:

$ less /var/log/cfn-init.log

If a specific package is causing issues, try reinstalling it:

$ sudo yum reinstall package-name

For dependency conflicts, you may need to remove conflicting packages:

$ sudo yum remove conflicting-package

If the issue persists, consider creating a custom AMI with your required packages pre-installed using the pcluster build-image command. For more information, see Amazon ParallelCluster AMI customization.

Check for errors in the chef-client log that mention specific configuration files:

$ grep -i "config" /var/log/chef-client.log

Look for permission or syntax errors in configuration files:

$ less /var/log/cfn-init.log

Restore modified configuration files to their original state:

$ sudo cp /etc/file.conf.bak /etc/file.conf

If you need to make persistent changes to system configuration files, use custom bootstrap actions instead of directly modifying files:

HeadNode:
  CustomActions:
    OnNodeConfigured:
      Script: s3://bucket-name/config-script.sh

For more information, see Custom bootstrap actions.

For configuration changes that must be made directly to system files, consider creating a custom AMI. For more information, see Amazon ParallelCluster AMI customization.

Check if kernel updates have been applied:

$ uname -r

Look for Amazon FSx mount failures in the logs:

$ grep -i "fsx" /var/log/chef-client.log

For Ubuntu 22.04, avoid updating to the latest kernel as there is no Amazon FSx client for that kernel. For more information, see Operating system considerations.

If you've already updated the kernel and are experiencing issues, consider downgrading to a compatible kernel version:

$ sudo apt install linux-image-previous-version

For persistent kernel customizations, create a custom AMI with the specific kernel version you need. For more information, see Amazon ParallelCluster AMI customization.

Use Custom Bootstrap Actions: Instead of directly modifying system files, use OnNodeStart or OnNodeConfigured scripts to make changes in a controlled manner. For more information, see Custom bootstrap actions.

Create Custom AMIs: For significant OS modifications, create a custom AMI using pcluster build-image rather than making changes to running instances. For more information, see Amazon ParallelCluster AMI customization.

Test Changes First: Before applying changes to a production cluster, test them on a small test cluster to ensure compatibility.

Document Changes: Keep track of all OS configuration changes made to facilitate troubleshooting.

Backup Configuration Files: Before modifying any system configuration file, create a backup:

$ sudo cp /etc/file.conf /etc/file.conf.bak

Check Logs After Changes: After making OS configuration changes, check the logs for any errors:

$ less /var/log/cfn-init.log
$ less /var/log/chef-client.log