Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Troubleshooting Patch Manager

Use the following information to help you troubleshoot problems with Patch Manager, a capability of Amazon Systems Manager.

Errors when running AWS-RunPatchBaseline on Linux

Issue: 'No such file or directory' error

Problem: When you run AWS-RunPatchBaseline, patching fails with one of the following errors.

IOError: [Errno 2] No such file or directory: 'patch-baseline-operations-X.XX.tar.gz'

Unable to extract tar file: /var/log/amazon/ssm/patch-baseline-operations/patch-baseline-operations-1.75.tar.gz.failed to run commands: exit status 155

Unable to load and extract the content of payload, abort.failed to run commands: exit status 152

Cause 1: Two commands to run AWS-RunPatchBaseline were running at the same time on the same managed node. This creates a race condition that results in the temporary file patch-baseline-operations* not being created or accessed properly.

Cause 2: Insufficient storage space remains under the /var directory.

Solution 1: Ensure that no maintenance window has two or more Run Command tasks that run AWS-RunPatchBaseline with the same Priority level and that run on the same target IDs. If this is the case, reorder the priority. Run Command is a capability of Amazon Systems Manager.

Solution 2: Ensure that only one maintenance window at a time is running Run Command tasks that use AWS-RunPatchBaseline on the same targets and on the same schedule. If this is the case, change the schedule.

Solution 3: Ensure that only one State Manager association is running AWS-RunPatchBaseline on the same schedule and targeting the same managed nodes. State Manager is a capability of Amazon Systems Manager.

Solution 4: Free up sufficient storage space under the /var directory for the update packages.

Issue: 'another process has acquired yum lock' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

12/20/2019 21:41:48 root [INFO]: another process has acquired yum lock, waiting 2 s and retry.

Cause: The AWS-RunPatchBaseline document has started running on a managed node where it's already running in another operation and and has acquired the package manager yum process.

Solution: Ensure that no State Manager association, maintenance window tasks, or other configurations that run AWS-RunPatchBaseline on a schedule) are targeting the same managed node around the same time.

Issue: 'Permission denied / failed to run commands' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

sh: 
/var/lib/amazon/ssm/instanceid/document/orchestration/commandid/PatchLinux/_script.sh: Permission denied
failed to run commands: exit status 126

Cause: /var/lib/amazon/ might be mounted with noexec permissions. This is an issue because SSM Agent downloads payload scripts to /var/lib/amazon/ssm and runs them from that location.

Solution: Ensure that you have have configured exclusive partitions to /var/log/amazon and /var/lib/amazon, and that they're mounted with exec permissions.

Issue: 'Unable to download payload' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

Unable to download payload: https://s3.DOC-EXAMPLE-BUCKET.region.amazonaws.com.cn/aws-ssm-region/patchbaselineoperations/linux/payloads/patch-baseline-operations-X.XX.tar.gz.failed to run commands: exit status 156

Cause: The managed node doesn't have the required permissions to access the specified Amazon Simple Storage Service (Amazon S3) bucket.

Solution: Update your network configuration so that S3 endpoints are reachable. For more details, see information about required access to S3 buckets for Patch Manager in SSM Agent communications with Amazon managed S3 buckets.

Issue: 'unsupported package manager and python version combination' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

An unsupported package manager and python version combination was found. Apt requires Python3 to be installed.
failed to run commands: exit status 1

Cause: python3 isn't installed on the Debian Server, Raspberry Pi OS, or Ubuntu Server instance.

Solution: Install python3 on the server, which is required for Debian Server, Raspberry Pi OS, and Ubuntu Server managed nodes.

Issue: Patch Manager isn't applying rules specified to exclude certain packages

Problem: You have attempted to exclude certain packages by specifying them in the /etc/yum.conf file, in the format exclude=package-name, but they aren't excluded during the Patch Manager Install operation.

Cause: Patch Manager doesn't incorporate exclusions specified in the /etc/yum.conf file.

Solution: To exclude specific packages, create a custom patch baseline and create a rule to exclude the packages you don't want installed.

Issue: Patching fails and Patch Manager reports that the Server Name Indication extension to TLS is not available

Problem: The patching operation issues the following message.

/var/log/amazon/ssm/patch-baseline-operations/urllib3/util/ssl_.py:369: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension
to TLS is not available on this platform. This may cause the server to present an incorrect TLS 
certificate, which can cause validation failures. You can upgrade to a newer version of Python 
to solve this. 
For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

Cause: This message doesn't indicate an error. Instead, it's a warning that the older version of Python distributed with the operating system doesn't support TLS Server Name Indication. The Systems Manager patch payload script issues this warning when connecting to Amazon APIs that support SNI.

Solution: To troubleshoot any patching failures when this message is reported, review the contents of the stdout and stderr files. If you haven't configured the patch baseline to store these files in an Amazon S3 bucket or in Amazon CloudWatch Logs, you can locate the files in the following location on your Linux managed node.

/var/lib/amazon/ssm/instance-id/document/orchestration/Run-Command-execution-id/awsrunShellScript/PatchLinux

Issue: Patch Manager reports 'No more mirrors to try'

Problem: The patching operation issues the following message.

[Errno 256] No more mirrors to try.

Cause: The repositories configured on the managed node are not working correctly. Possible causes for this include:

Solution: Patch Manager uses the managed node’s default package manager to perform patching operation. Double-check that repositories are configured and operating correctly.

Errors when running AWS-RunPatchBaseline on Windows Server

Issue: mismatched product family/product pairs

Problem: When you create a patch baseline in the Systems Manager console, you specify a product family and a product. For example, you might choose:

Cause: If you attempt to create a patch baseline with a mismatched product family/product pair, an error message is displayed. The following are reasons this can occur:

Solution: To avoid this issue in the console, always choose options from the Currently available options sublists.

You can also view the products that have available patches by using the describe-patch-properties command in the Amazon CLI or the DescribePatchProperties API command.

Issue: AWS-RunPatchBaseline output returns an HRESULT (Windows Server)

Problem: You received an error like the following.

----------ERROR-------
Invoke-PatchBaselineOperation : Exception Details: An error occurred when 
attempting to search Windows Update.
Exception Level 1:
 Error Message: Exception from HRESULT: 0x80240437
 Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)..
(Windows updates)
11/22/2020 09:17:30 UTC | Info | Searching for Windows Updates.
11/22/2020 09:18:59 UTC | Error | Searching for updates resulted in error: Exception from HRESULT: 0x80240437
----------ERROR-------
failed to run commands: exit status 4294967295

Cause: This output indicates that the native Windows Update APIs were unable to run the patching operations.

Solution: Check the HResult code in the Microsoft documentation to identify troubleshooting steps for resolving the error.

Issue: managed node doesn't have access to Windows Update Catalog or WSUS

Problem: You received an error like the following.

Downloading PatchBaselineOperations PowerShell module from /path_to_module.zip to C:\Windows\TEMP\Amazon.PatchBaselineOperations-1.29.zip.

Extracting PatchBaselineOperations zip file contents to temporary folder.

Verifying SHA 256 of the PatchBaselineOperations PowerShell module files.

Successfully downloaded and installed the PatchBaselineOperations PowerShell module.

Patch Summary for

PatchGroup :

BaselineId :

Baseline : null

SnapshotId :

RebootOption : RebootIfNeeded

OwnerInformation :

OperationType : Scan

OperationStartTime : 1970-01-01T00:00:00.0000000Z

OperationEndTime : 1970-01-01T00:00:00.0000000Z

InstalledCount : -1

InstalledRejectedCount : -1

InstalledPendingRebootCount : -1

InstalledOtherCount : -1

FailedCount : -1

MissingCount : -1

NotApplicableCount : -1

UnreportedNotApplicableCount : -1

EC2AMAZ-VL3099P - PatchBaselineOperations Assessment Results - 2020-12-30T20:59:46.169

----------ERROR-------

Invoke-PatchBaselineOperation : Exception Details: An error occurred when attempting to search Windows Update.

Exception Level 1:

Error Message: Exception from HRESULT: 0x80072EE2

Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)

at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String

searchCriteria)

At C:\ProgramData\Amazon\SSM\InstanceData\i-02573cafcfEXAMPLE\document\orchestration\3d2d4864-04b7-4316-84fe-eafff1ea58

e3\PatchWindows\_script.ps1:230 char:13

+ $response = Invoke-PatchBaselineOperation -Operation Install -Snapsho ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : OperationStopped: (Amazon.Patch.Ba...UpdateOperation:InstallWindowsUpdateOperation) [Inv

oke-PatchBaselineOperation], Exception

+ FullyQualifiedErrorId : Exception Level 1:

Error Message: Exception Details: An error occurred when attempting to search Windows Update.

Exception Level 1:

Error Message: Exception from HRESULT: 0x80072EE2

Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)

at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String searc

---Error truncated----

Cause: This error could be related to the Windows Update components, or to a lack of connectivity to the Windows Update Catalog or Windows Server Update Services (WSUS).

Solution: Confirm that the managed node has connectivity to the Microsoft Update Catalog through an internet gateway, NAT gateway, or NAT instance. If you're using WSUS, confirm that the managed node has connectivity to the WSUS server in your environment. If connectivity is available to the intended destination, check the Microsoft documentation for other potential causes of HResult 0x80072EE2. This might indicate an operating system level issue.

Issue: PatchBaselineOperations PowerShell module is not downloadable

Problem: You received an error like the following.

Preparing to download PatchBaselineOperations PowerShell module from S3.
                    
Downloading PatchBaselineOperations PowerShell module from /path_to_module.zip to C:\Windows\TEMP\Amazon.PatchBaselineOperations-1.29.zip.
----------ERROR-------

C:\ProgramData\Amazon\SSM\InstanceData\i-02573cafcfEXAMPLE\document\orchestration\aaaaaaaa-bbbb-cccc-dddd-4f6ed6bd5514\

PatchWindows\_script.ps1 : An error occurred when executing PatchBaselineOperations: Unable to connect to the remote server

+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException

+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,_script.ps1

failed to run commands: exit status 4294967295

Solution: Check the managed node connectivity and permissions to Amazon Simple Storage Service (Amazon S3). The managed node's Amazon Identity and Access Management (IAM) role must use the minimum permissions cited in SSM Agent communications with Amazon managed S3 buckets. The node must communicate with the Amazon S3 endpoint through the Amazon S3 gateway endpoint, NAT gateway, or internet gateway. For more information about the VPC Endpoint requirements for Amazon Systems Manager SSM Agent (SSM Agent), see Step 6: (Optional) Create a VPC endpoint.

Issue: missing patches

Problem: AWS-RunPatchbaseline completed successfully, but there are some missing patches.

The following are some common causes and their solutions.

Cause 1: The baseline isn't effective.

Solution 1: To check if this is the cause, use the following procedure.

Cause 2: The patch was replaced.

Solution 2: To check if this is true, use the following procedure.

Cause 3: The same patch might have different KB numbers because the WSUS and Window online updates are handled as independent Release Channels by Microsoft.

Solution 3: Check the patch eligibility. If the package isn't available under WSUS, install OS Build 14393.3115. If the package is available for all operating system builds, install OS Builds 18362.1256 and 18363.1256.

Contacting Amazon Web Services Support

If you can't find troubleshooting solutions in this section, and you have a Developer, Business, or Enterprise Amazon Web Services Support plan, you can create a technical support case at Amazon Web Services Support.

Before you contact Amazon Web Services Support, collect the following items: