# Troubleshooting a custom key store
Amazon CloudHSM key stores are designed to be available and resilient. However, there are some error conditions that you might have to repair to keep your Amazon CloudHSM key store operational.
**Topics**
+ [How to fix unavailable KMS keys](#fix-unavailable-cmks)
+ [How to fix a failing KMS key](#fix-cmk-failed)
+ [How to fix a connection failure](#fix-keystore-failed)
+ [How to respond to a cryptographic operation failure](#fix-keystore-communication)
+ [How to fix invalid `kmsuser` credentials](#fix-keystore-password)
+ [How to delete orphaned key material](#fix-keystore-orphaned-key)
+ [How to recover deleted key material for a KMS key](#fix-keystore-recover-backing-key)
+ [How to log in as `kmsuser`](#fix-login-as-kmsuser)
## How to fix unavailable KMS keys
The [key state](key-state.md) of Amazon KMS keys in an Amazon CloudHSM key store is typically `Enabled`. Like all KMS keys, the key state changes when you disable the KMS keys in an Amazon CloudHSM key store or schedule them for deletion. However, unlike other KMS keys, the KMS keys in a custom key store can also have a [key state](key-state.md) of `Unavailable`.
A key state of `Unavailable` indicates that the KMS key is in a custom key store that was intentionally [disconnected](disconnect-keystore.md) and attempts to reconnect it, if any, failed. While a KMS key is unavailable, you can view and manage the KMS key, but you cannot use it for [cryptographic operations](manage-cmk-keystore.md#use-cmk-keystore).
To find the key state of a KMS key, on the **Customer managed keys** page, view the **Status** field of the KMS key. Or, use the [DescribeKey](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeKey.html) operation and view the `KeyState` element in the response. For details, see [Identify and view keys](viewing-keys.md).
The KMS keys in a disconnected custom key store will have a key state of `Unavailable` or `PendingDeletion`. KMS keys that are scheduled for deletion from a custom key store have a `Pending Deletion` key state, even when the custom key store is disconnected. This allows you to cancel the scheduled key deletion without reconnecting the custom key store.
To fix an unavailable KMS key, [reconnect the custom key store](disconnect-keystore.md). After the custom key store is reconnected, the key state of the KMS keys in the custom key store is automatically restored to its previous state, such as `Enabled` or `Disabled`. KMS keys that are pending deletion remain in the `PendingDeletion` state. However, while the problem persists, [enabling and disabling an unavailable KMS key](enabling-keys.md) does not change its key state. The enable or disable action takes effect only when the key becomes available.
For help with failed connections, see [How to fix a connection failure](#fix-keystore-failed).
## How to fix a failing KMS key
Problems with creating and using KMS keys in Amazon CloudHSM key stores can be caused by a problem with your Amazon CloudHSM key store, its associated Amazon CloudHSM cluster, the KMS key, or its key material.
When an Amazon CloudHSM key store is disconnected from its Amazon CloudHSM cluster, the key state of KMS keys in the custom key store is `Unavailable`. All requests to create KMS keys in a disconnected Amazon CloudHSM key store return a `CustomKeyStoreInvalidStateException` exception. All requests to encrypt, decrypt, re-encrypt, or generate data keys return a `KMSInvalidStateException` exception. To fix the problem, [reconnect the Amazon CloudHSM key store](connect-keystore.md).
However, your attempts to use a KMS key in an Amazon CloudHSM key store for [cryptographic operations](manage-cmk-keystore.md#use-cmk-keystore) might fail even when its key state is `Enabled` and the connection state of the Amazon CloudHSM key store is `Connected`. This might be caused by any of the following conditions.
+ The key material for the KMS key might have been deleted from the associated Amazon CloudHSM cluster. To investigate, [find the key id](find-handle-for-cmk-id.md) of the key material for a KMS key and, if necessary, try to [recover the key material](#fix-keystore-recover-backing-key).
+ All HSMs were deleted from the Amazon CloudHSM cluster that is associated with the Amazon CloudHSM key store. To use a KMS key in an Amazon CloudHSM key store in a cryptographic operation, its Amazon CloudHSM cluster must contain at least one active HSM. To verify the number and state of HSMs in an Amazon CloudHSM cluster, [use the Amazon CloudHSM console](https://docs.amazonaws.cn/cloudhsm/latest/userguide/add-remove-hsm.html) or the [DescribeClusters](https://docs.amazonaws.cn/cloudhsm/latest/APIReference/API_DescribeClusters.html) operation. To add an HSM to the cluster, use the Amazon CloudHSM console or the [CreateHsm](https://docs.amazonaws.cn/cloudhsm/latest/APIReference/API_CreateHsm.html) operation.
+ The Amazon CloudHSM cluster associated with the Amazon CloudHSM key store was deleted. To fix the problem, [create a cluster from a backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html) that is related to the original cluster, such as a backup of the original cluster, or a backup that was used to create the original cluster. Then, [edit the cluster ID](update-keystore.md) in the custom key store settings. For instructions, see [How to recover deleted key material for a KMS key](#fix-keystore-recover-backing-key).
+ The Amazon CloudHSM cluster associated with the custom key store did not have any available PKCS \#11 sessions. This typically occurs during periods of high burst traffic when additional sessions are needed to service the traffic. To respond to a `KMSInternalException` with an error message about PKCS \#11 sessions, back off and retry the request again.
## How to fix a connection failure
If you try to [connect an Amazon CloudHSM key store](connect-keystore.md) to its Amazon CloudHSM cluster, but the operation fails, the connection state of the Amazon CloudHSM key store changes to `FAILED`. To find the connection state of an Amazon CloudHSM key store, use the Amazon KMS console or the [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) operation.
Alternatively, some connection attempts fail quickly due to easily detected cluster configuration errors. In this case, the connection state is still `DISCONNECTED`. These failures return an error message or [exception](https://docs.amazonaws.cn/kms/latest/APIReference/API_ConnectCustomKeyStore.html#API_ConnectCustomKeyStore_Errors) that explains why the attempt failed. Review the exception description and [cluster requirements](create-keystore.md#before-keystore), fix the problem, [update the Amazon CloudHSM key store](update-keystore.md), if necessary, and try to connect again.
When the connection state is `FAILED`, run the [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) operation and see the `ConnectionErrorCode` element in the response.
**Note**
When the connection state of an Amazon CloudHSM key store is `FAILED`, you must [disconnect the Amazon CloudHSM key store](disconnect-keystore.md) before attempting to reconnect it. You cannot connect an Amazon CloudHSM key store with a `FAILED` connection state.
+ `CLUSTER_NOT_FOUND` indicates that Amazon KMS cannot find an Amazon CloudHSM cluster with the specified cluster ID. This might occur because the wrong cluster ID was provided to an API operation or the cluster was deleted and not replaced. To fix this error, verify the cluster ID, such as by using the Amazon CloudHSM console or the [DescribeClusters](https://docs.amazonaws.cn/cloudhsm/latest/APIReference/API_DescribeClusters.html) operation. If the cluster was deleted, [create a cluster from a recent backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html) of the original. Then, [disconnect the Amazon CloudHSM key store](disconnect-keystore.md), [edit the Amazon CloudHSM key store](update-keystore.md) cluster ID setting, and [reconnect the Amazon CloudHSM key store](connect-keystore.md) to the cluster.
+ `INSUFFICIENT_CLOUDHSM_HSMS` indicates that the associated Amazon CloudHSM cluster does not contain any HSMs. To connect, the cluster must have at least one HSM. To find the number of HSMs in the cluster, use the [DescribeClusters](https://docs.amazonaws.cn/cloudhsm/latest/APIReference/API_DescribeClusters.html) operation. To resolve this error, [add at least one HSM](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-hsm.html) to the cluster. If you add multiple HSMs, it's best to create them in different Availability Zones.
+ `INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET` indicates that Amazon KMS could not connect the Amazon CloudHSM key store to its Amazon CloudHSM cluster because at least one [private subnet associated with the cluster](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-subnets.html) doesn't have any available IP addresses. An Amazon CloudHSM key store connection requires one free IP address in each of the associated private subnets, although two are preferable.
You [can't add IP addresses](http://www.amazonaws.cn/support-plans/knowledge-center/vpc-ip-address-range/) (CIDR blocks) to an existing subnet. If possible, move or delete other resources that are using the IP addresses in the subnet, such as unused EC2 instances or elastic network interfaces. Otherwise, you can [create a cluster from a recent backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html) of the Amazon CloudHSM cluster with new or existing private subnets that have [more free address space](https://docs.amazonaws.cn/vpc/latest/userguide/configure-subnets.html#subnet-sizing). Then, to associate the new cluster with your Amazon CloudHSM key store, [disconnect the custom key store](disconnect-keystore.md), [change the cluster ID](update-keystore.md) of the Amazon CloudHSM key store to the ID of the new cluster, and try to connect again.
**Tip**
To avoid [resetting the `kmsuser` password](#fix-keystore-password), use the most recent backup of the Amazon CloudHSM cluster.
+ `INTERNAL_ERROR` indicates that Amazon KMS could not complete the request due to an internal error. Retry the request. For `ConnectCustomKeyStore` requests, disconnect the Amazon CloudHSM key store before trying to connect again.
+ `INVALID_CREDENTIALS` indicates that Amazon KMS cannot log into the associated Amazon CloudHSM cluster because it doesn't have the correct `kmsuser` account password. For help with this error, see [How to fix invalid `kmsuser` credentials](#fix-keystore-password).
+ `NETWORK_ERRORS` usually indicates transient network issues. [Disconnect the Amazon CloudHSM key store](disconnect-keystore.md), wait a few minutes, and try to connect again.
+ `SUBNET_NOT_FOUND` indicates that at least one subnet in the Amazon CloudHSM cluster configuration was deleted. If Amazon KMS cannot find all of the subnets in the cluster configuration, attempts to connect the Amazon CloudHSM key store to the Amazon CloudHSM cluster fail.
To fix this error, [create a cluster from a recent backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html) of the same Amazon CloudHSM cluster. (This process creates a new cluster configuration with a VPC and private subnets.) Verify that the new cluster meets the [requirements for a custom key store](create-keystore.md#before-keystore), and note the new cluster ID. Then, to associate the new cluster with your Amazon CloudHSM key store, [disconnect the custom key store](disconnect-keystore.md), [change the cluster ID](update-keystore.md) of the Amazon CloudHSM key store to the ID of the new cluster, and try to connect again.
**Tip**
To avoid [resetting the `kmsuser` password](#fix-keystore-password), use the most recent backup of the Amazon CloudHSM cluster.
+ `USER_LOCKED_OUT` indicates that the [`kmsuser` crypto user (CU) account](keystore-cloudhsm.md#concept-kmsuser) is locked out of the associated Amazon CloudHSM cluster due to too many failed password attempts. For help with this error, see [How to fix invalid `kmsuser` credentials](#fix-keystore-password).
To fix this error, [disconnect the Amazon CloudHSM key store](disconnect-keystore.md) and use the [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html) command in CloudHSM CLI to change the `kmsuser` account password. Then, [edit the `kmsuser` password setting](update-keystore.md) for the custom key store, and try to connect again. For help, use the procedure described in the [How to fix invalid `kmsuser` credentials](#fix-keystore-password) topic.
+ `USER_LOGGED_IN` indicates that the `kmsuser` CU account is logged into the associated Amazon CloudHSM cluster. This prevents Amazon KMS from rotating the `kmsuser` account password and logging into the cluster. To fix this error, log the `kmsuser` CU out of the cluster. If you changed the `kmsuser` password to log into the cluster, you must also and update the key store password value for the Amazon CloudHSM key store. For help, see [How to log out and reconnect](#login-kmsuser-2).
+ `USER_NOT_FOUND` indicates that Amazon KMS cannot find a `kmsuser` CU account in the associated Amazon CloudHSM cluster. To fix this error, [create a `kmsuser` CU account](create-keystore.md#kmsuser-concept) in the cluster, and then [update the key store password value](update-keystore.md) for the Amazon CloudHSM key store. For help, see [How to fix invalid `kmsuser` credentials](#fix-keystore-password).
## How to respond to a cryptographic operation failure
A cryptographic operation that uses a KMS key in a custom key store might fail with a `KMSInvalidStateException`. The following error messages might accompany the `KMSInvalidStateException`.
| |
| --- |
| KMS cannot communicate with your CloudHSM cluster. This might be a transient network issue. If you see this error repeatedly, verify that the Network ACLs and the security group rules for the VPC of your Amazon CloudHSM cluster are correct. |
+ Although this is an HTTPS 400 error, it might result from transient network issues. To respond, begin by retrying the request. However, if it continues to fail, examine the configuration of your networking components. This error is most likely caused by the misconfiguration of a networking component, such as a firewall rule or VPC security group rule that is blocking outgoing traffic. For example, KMS cannot communicate with Amazon CloudHSM clusters over IPv6. For details on prerequisites, see [Create an Amazon CloudHSM key store](create-keystore.md).
| |
| --- |
| KMS cannot communicate with your Amazon CloudHSM cluster because the kmsuser is locked out. If you see this error repeatedly, disconnect the Amazon CloudHSM key store and reset the kmsuser account password. Update the kmsuser password for the custom key store and try the request again. |
+ This error message indicates that the [`kmsuser` crypto user (CU) account](keystore-cloudhsm.md#concept-kmsuser) is locked out of the associated Amazon CloudHSM cluster due to too many failed password attempts. For help with this error, see [How to disconnect and log in](#login-kmsuser-1).
## How to fix invalid `kmsuser` credentials
When you [connect an Amazon CloudHSM key store](connect-keystore.md), Amazon KMS logs into the associated Amazon CloudHSM cluster as the [`kmsuser` crypto user](keystore-cloudhsm.md#concept-kmsuser) (CU). It remains logged in until the Amazon CloudHSM key store is disconnected. The [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) response shows a `ConnectionState` of `FAILED` and `ConnectionErrorCode` value of `INVALID_CREDENTIALS`, as shown in the following example.
If you disconnect the Amazon CloudHSM key store and change the `kmsuser` password, Amazon KMS cannot log into the Amazon CloudHSM cluster with the credentials of the `kmsuser` CU account. As a result, all attempts to connect the Amazon CloudHSM key store fail. The `DescribeCustomKeyStores` response shows a `ConnectionState` of `FAILED` and `ConnectionErrorCode` value of `INVALID_CREDENTIALS`, as shown in the following example.
```
$ aws kms describe-custom-key-stores --custom-key-store-name {{ExampleKeyStore}}
{
"CustomKeyStores": [
"CloudHsmClusterId": "cluster-1a23b4cdefg",
"ConnectionErrorCode": "INVALID_CREDENTIALS"
"CustomKeyStoreId": "cks-1234567890abcdef0",
"CustomKeyStoreName": "ExampleKeyStore",
"TrustAnchorCertificate": "{{}}",
"CreationDate": "1.499288695918E9",
"ConnectionState": "FAILED"
],
}
```
Also, after five failed attempts to log into the cluster with an incorrect password, Amazon CloudHSM locks the user account. To log into the cluster, you must change the account password.
If Amazon KMS gets a lockout response when it tries to log into the cluster as the `kmsuser` CU, the request to connect the Amazon CloudHSM key store fails. The [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) response includes a `ConnectionState` of `FAILED` and `ConnectionErrorCode` value of `USER_LOCKED_OUT`, as shown in the following example.
```
$ aws kms describe-custom-key-stores --custom-key-store-name {{ExampleKeyStore}}
{
"CustomKeyStores": [
"CloudHsmClusterId": "cluster-1a23b4cdefg",
"ConnectionErrorCode": "USER_LOCKED_OUT"
"CustomKeyStoreId": "cks-1234567890abcdef0",
"CustomKeyStoreName": "ExampleKeyStore",
"TrustAnchorCertificate": "{{}}",
"CreationDate": "1.499288695918E9",
"ConnectionState": "FAILED"
],
}
```
To repair any of these conditions, use the following procedure.
1. [Disconnect the Amazon CloudHSM key store](disconnect-keystore.md).
1. Run the [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) operation and view the value of the `ConnectionErrorCode` element in the response.
+ If the `ConnectionErrorCode` value is `INVALID_CREDENTIALS`, determine the current password for the `kmsuser` account. If necessary, use the [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html) command in CloudHSM CLI to set the password to a known value.
+ If the `ConnectionErrorCode` value is `USER_LOCKED_OUT`, you must use the [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html) command in CloudHSM CLI to change the `kmsuser` password.
1. [Edit the `kmsuser` password setting](update-keystore.md) so it matches the current `kmsuser` password in the cluster. This action tells Amazon KMS which password to use to log into the cluster. It does not change the `kmsuser` password in the cluster.
1. [Connect the custom key store](connect-keystore.md).
## How to delete orphaned key material
After scheduling deletion of a KMS key from an Amazon CloudHSM key store, you might need to manually delete the corresponding key material from the associated Amazon CloudHSM cluster.
When you create a KMS key in an Amazon CloudHSM key store, Amazon KMS creates the KMS key metadata in Amazon KMS and generates the key material in the associated Amazon CloudHSM cluster. When you schedule deletion of a KMS key in an Amazon CloudHSM key store, after the waiting period, Amazon KMS deletes the KMS key metadata. Then Amazon KMS makes a best effort to delete the corresponding key material from the Amazon CloudHSM cluster. The attempt might fail if Amazon KMS cannot access the cluster, such as when it's disconnected from the Amazon CloudHSM key store or the `kmsuser` password changes. Amazon KMS does not attempt to delete key material from cluster backups.
Amazon KMS reports the results of its attempt to delete the key material from the cluster in the `DeleteKey` event entry of your Amazon CloudTrail logs. It appears in the `backingKeysDeletionStatus` element of the `additionalEventData` element, as shown in the following example entry. The entry also includes the KMS key ARN, the Amazon CloudHSM cluster ID, and the ID (`backing-key-id`) of the key material.
```
{
"eventVersion": "1.08",
"userIdentity": {
"accountId": "111122223333",
"invokedBy": "Amazon Internal"
},
"eventTime": "2021-12-10T14:23:51Z",
"eventSource": "kms.amazonaws.com",
"eventName": "DeleteKey",
"awsRegion": "us-west-2",
"sourceIPAddress": "Amazon Internal",
"userAgent": "AWS Internal",
"requestParameters": null,
"responseElements": {
"keyId":"arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
},
"additionalEventData": {
"customKeyStoreId": "cks-1234567890abcdef0",
"clusterId": "cluster-1a23b4cdefg",
"backingKeys": "[{\"backingKeyId\":\"{{backing-key-id}}\"}]",
"backingKeysDeletionStatus": "[{\"backingKeyId\":\"{{backing-key-id}}\",\"deletionStatus\":\"FAILURE\"}]"
},
"eventID": "c21f1f47-f52b-4ffe-bff0-6d994403cf40",
"readOnly": false,
"resources": [
{
"accountId": "111122223333",
"type": "AWS::KMS::Key",
"ARN": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}
],
"eventType": "AwsServiceEvent",
"recipientAccountId": "111122223333",
"managementEvent": true,
"eventCategory": "Management"
}
```
**Notes**
The following procedures use the Amazon CloudHSM Client SDK 5 command line tool, [CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli.html). The CloudHSM CLI replaces `key-handle` with `key-reference`.
On January 1, 2025, Amazon CloudHSM will end support for the Client SDK 3 command line tools, the CloudHSM Management Utility (CMU) and the Key Management Utility (KMU). For more information on the differences between the Client SDK 3 command line tools and the Client SDK 5 command line tool, see [Migrate from Client SDK 3 CMU and KMU to Client SDK 5 CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-migrate-from-kmu-cmu.html) in the *Amazon CloudHSM User Guide*.
The following procedures demonstrate how to delete the orphaned key material from the associated Amazon CloudHSM cluster.
1. Disconnect the Amazon CloudHSM key store, if it is not already disconnected, then [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-login.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-login.html), as explained in [How to disconnect and log in](#login-kmsuser-1).
**Note**
While a custom key store is disconnected, all attempts to create KMS keys in the custom key store or to use existing KMS keys in cryptographic operations will fail. This action can prevent users from storing and accessing sensitive data.
1. Use the [key delete](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-key-delete.html) command in CloudHSM CLI to delete the key from the HSMs in the cluster.
All CloudTrail log entries for cryptographic operation with a KMS key in a Amazon CloudHSM key store include an `additionalEventData` field with the `customKeyStoreId` and `backingKey`. The value returned in the `backingKeyId` field is the CloudHSM key `id` attribute. We recommend filtering the **key delete** operation by `id` to delete the orphaned key material you identified in your CloudTrail logs.
Amazon CloudHSM recognizes the `backingKeyId` value as a hexadecimal value. To filter by `id`, you must prepend the `backingKeyId` with `Ox`. For example, if the `backingKeyId` in your CloudTrail log is `1a2b3c45678abcdef`, you would filter by `0x1a2b3c45678abcdef`.
The following example deletes a key from the HSMs in your cluster. The `backing-key-id` is listed in the CloudTrail log entry. Before running this command, replace the example `backing-key-id` with a valid one from your account.
```
aws-cloudhsm key delete --filter attr.id="{{0x}}"
{
"error_code": 0,
"data": {
"message": "Key deleted successfully"
}
}
```
1. Log out and reconnect the Amazon CloudHSM key store as described in [How to log out and reconnect](#login-kmsuser-2).
## How to recover deleted key material for a KMS key
If the key material for an Amazon KMS key is deleted, the KMS key is unusable and all ciphertext that was encrypted under the KMS key cannot be decrypted. This can happen if the key material for a KMS key in an Amazon CloudHSM key store is deleted from the associated Amazon CloudHSM cluster. However, it might be possible to recover the key material.
When you create an Amazon KMS key (KMS key) in an Amazon CloudHSM key store, Amazon KMS logs into the associated Amazon CloudHSM cluster and creates the key material for the KMS key. It also changes the password to a value that only it knows and remains logged in as long as the Amazon CloudHSM key store is connected. Because only the key owner, that is, the CU who created a key, can delete the key, it is unlikely that the key will be deleted from the HSMs accidentally.
However, if the key material for a KMS key is deleted from the HSMs in a cluster, the key state of the KMS key eventually changes to `UNAVAILABLE`. If you attempt to use the KMS key for a cryptographic operation, the operation fails with a `KMSInvalidStateException` exception. Most importantly, any data that was encrypted under the KMS key cannot be decrypted.
Under certain circumstances, you can recover deleted key material by [creating a cluster from a backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html) that contains the key material. This strategy works only when at least one backup was created while the key existed and before it was deleted.
Use the following process to recover the key material.
1. Find a cluster backup that contains the key material. The backup must also contain all users and keys that you need to support the cluster and its encrypted data.
Use the [DescribeBackups](https://docs.amazonaws.cn/cloudhsm/latest/APIReference/API_DescribeBackups.html) operation to list the backups for a cluster. Then use the backup timestamp to help you select a backup. To limit the output to the cluster that is associated with the Amazon CloudHSM key store, use the `Filters` parameter, as shown in the following example.
```
$ aws cloudhsmv2 describe-backups --filters clusterIds={{}}
{
"Backups": [
{
"ClusterId": "cluster-1a23b4cdefg",
"BackupId": "backup-9g87f6edcba",
"CreateTimestamp": 1536667238.328,
"BackupState": "READY"
},
...
]
}
```
1. [Create a cluster from the selected backup](https://docs.amazonaws.cn/cloudhsm/latest/userguide/create-cluster-from-backup.html). Verify that the backup contains the deleted key and other users and keys that the cluster requires.
1. [Disconnect the Amazon CloudHSM key store](disconnect-keystore.md) so you can edit its properties.
1. [Edit the cluster ID](update-keystore.md) of the Amazon CloudHSM key store. Enter the cluster ID of the cluster that you created from the backup. Because the cluster shares a backup history with the original cluster, the new cluster ID should be valid.
1. [Reconnect the Amazon CloudHSM key store](connect-keystore.md).
## How to log in as `kmsuser`
To create and manage the key material in the Amazon CloudHSM cluster for your Amazon CloudHSM key store, Amazon KMS uses the [`kmsuser` crypto user (CU) account](keystore-cloudhsm.md#concept-kmsuser). You [create the `kmsuser` CU account](create-keystore.md#before-keystore) in your cluster and provide its password to Amazon KMS when you create your Amazon CloudHSM key store.
In general, Amazon KMS manages the `kmsuser` account. However, for some tasks, you need to disconnect the Amazon CloudHSM key store, log into the cluster as the `kmsuser` CU, and use the [CloudHSM Command Line Interface (CLI)](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli.html).
**Note**
While a custom key store is disconnected, all attempts to create KMS keys in the custom key store or to use existing KMS keys in cryptographic operations will fail. This action can prevent users from storing and accessing sensitive data.
This topic explains how to [disconnect your Amazon CloudHSM key store and log in](#login-kmsuser-1) as `kmsuser`, run the Amazon CloudHSM command line tool, and [log out and reconnect your Amazon CloudHSM key store](#login-kmsuser-2).
**Topics**
+ [How to disconnect and log in](#login-kmsuser-1)
+ [How to log out and reconnect](#login-kmsuser-2)
### How to disconnect and log in
Use the following procedure each time to need to log into an associated cluster as the `kmsuser` crypto user.
**Notes**
The following procedures use the Amazon CloudHSM Client SDK 5 command line tool, [CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli.html). The CloudHSM CLI replaces `key-handle` with `key-reference`.
On January 1, 2025, Amazon CloudHSM will end support for the Client SDK 3 command line tools, the CloudHSM Management Utility (CMU) and the Key Management Utility (KMU). For more information on the differences between the Client SDK 3 command line tools and the Client SDK 5 command line tool, see [Migrate from Client SDK 3 CMU and KMU to Client SDK 5 CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-migrate-from-kmu-cmu.html) in the *Amazon CloudHSM User Guide*.
1. Disconnect the Amazon CloudHSM key store, if it is not already disconnected. You can use the Amazon KMS console or Amazon KMS API.
While your Amazon CloudHSM key is connected, Amazon KMS is logged in as the `kmsuser`. This prevents you from logging in as `kmsuser` or changing the `kmsuser` password.
For example, this command uses [DisconnectCustomKeyStore](https://docs.amazonaws.cn/kms/latest/APIReference/API_DisconnectCustomKeyStore.html) to disconnect an example key store. Replace the example Amazon CloudHSM key store ID with a valid one.
```
$ aws kms disconnect-custom-key-store --custom-key-store-id {{cks-1234567890abcdef0}}
```
1. Use the **login** command to login as an admin. Use the procedures described in the [Using CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-getting-started.html#w17aac19c11c13b7) section of the *Amazon CloudHSM User Guide*.
```
aws-cloudhsm > login --username admin --role admin
Enter password:
{
"error_code": 0,
"data": {
"username": "admin",
"role": "admin"
}
}
```
1. Use the [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-user-change-password.html) command in CloudHSM CLI to change the password of the `kmsuser` account to one that you know. (Amazon KMS rotates the password when you connect your Amazon CloudHSM key store.) The password must consist of 7-32 alphanumeric characters. It is case-sensitive and cannot contain any special characters.
1. Login as `kmsuser` using the password that you set. For detailed instructions, see the [Using CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-getting-started.html#w17aac19c11c13b7) section of the *Amazon CloudHSM User Guide*.
```
aws-cloudhsm > login --username kmsuser --role crypto-user
Enter password:
{
"error_code": 0,
"data": {
"username": "kmsuser",
"role": "crypto-user"
}
}
```
### How to log out and reconnect
Use the following procedure each time you need to log out as the `kmsuser` crypto user and reconnect your key store.
**Notes**
The following procedures use the Amazon CloudHSM Client SDK 5 command line tool, [CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli.html). The CloudHSM CLI replaces `key-handle` with `key-reference`.
On January 1, 2025, Amazon CloudHSM will end support for the Client SDK 3 command line tools, the CloudHSM Management Utility (CMU) and the Key Management Utility (KMU). For more information on the differences between the Client SDK 3 command line tools and the Client SDK 5 command line tool, see [Migrate from Client SDK 3 CMU and KMU to Client SDK 5 CloudHSM CLI](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-migrate-from-kmu-cmu.html) in the *Amazon CloudHSM User Guide*.
1. Perform the task, then use the [https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-logout.html](https://docs.amazonaws.cn/cloudhsm/latest/userguide/cloudhsm_cli-logout.html) command in CloudHSM CLI to log out. If you do not log out, attempts to reconnect your Amazon CloudHSM key store will fail.
```
aws-cloudhsm logout
{
"error_code": 0,
"data": "Logout successful"
}
```
1. [Edit the `kmsuser` password setting](update-keystore.md) for the custom key store.
This tells Amazon KMS the current password for `kmsuser` in the cluster. If you omit this step, Amazon KMS will not be able to log into the cluster as `kmsuser`, and all attempts to reconnect your custom key store will fail. You can use the Amazon KMS console or the `KeyStorePassword` parameter of the [UpdateCustomKeyStore](https://docs.amazonaws.cn/kms/latest/APIReference/API_UpdateCustomKeyStore.html) operation.
For example, this command tells Amazon KMS that the current password is `tempPassword`. Replace the example password with the actual one.
```
$ aws kms update-custom-key-store --custom-key-store-id {{cks-1234567890abcdef0}} --key-store-password {{tempPassword}}
```
1. Reconnect the Amazon KMS key store to its Amazon CloudHSM cluster. Replace the example Amazon CloudHSM key store ID with a valid one. During the connection process, Amazon KMS changes the `kmsuser` password to a value that only it knows.
The [ConnectCustomKeyStore](https://docs.amazonaws.cn/kms/latest/APIReference/API_ConnectCustomKeyStore.html) operation returns quickly, but the connection process can take an extended period of time. The initial response does not indicate the success of the connection process.
```
$ aws kms connect-custom-key-store --custom-key-store-id {{cks-1234567890abcdef0}}
```
1. Use the [DescribeCustomKeyStores](https://docs.amazonaws.cn/kms/latest/APIReference/API_DescribeCustomKeyStores.html) operation to verify that the Amazon CloudHSM key store is connected. Replace the example Amazon CloudHSM key store ID with a valid one.
In this example, the connection state field shows that the Amazon CloudHSM key store is now connected.
```
$ aws kms describe-custom-key-stores --custom-key-store-id {{cks-1234567890abcdef0}}
{
"CustomKeyStores": [
"CustomKeyStoreId": "cks-1234567890abcdef0",
"CustomKeyStoreName": "ExampleKeyStore",
"CloudHsmClusterId": "cluster-1a23b4cdefg",
"TrustAnchorCertificate": "{{}}",
"CreationDate": "1.499288695918E9",
"ConnectionState": "CONNECTED"
],
}
```