View a markdown version of this page

Validating, cutting over, and rolling back during a clone-based upgrade - Amazon OpenSearch Service
Monitor progressValidate the cloned clusterCut over to the cloned clusterCancel the upgrade before cutoverRetention periodRoll backChargesTroubleshooting
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Validating, cutting over, and rolling back during a clone-based upgrade

After you start a clone-based upgrade, OpenSearch Service creates a cloned cluster of your domain and copies your data to it. When the cloned cluster is ready, the upgrade waits for you. This page explains how to validate the cloned cluster, cut over to it, cancel the upgrade, and roll back. For information about the requests that perform each action, see Upgrading a domain (CLI) and Upgrading a domain (SDK).

Monitor progress

To track a clone-based upgrade, call DescribeDomainChangeProgress. The response lists the following stages. The stages refer to the cloned cluster as the new environment and to the primary cluster as the old environment.

Stage What it means
Initializing OpenSearch Service initializes the upgrade.
Validation OpenSearch Service checks that your domain is eligible for the upgrade.
Creating a new environment OpenSearch Service creates the cloned cluster.
Provisioning new nodes OpenSearch Service creates the cloned cluster's nodes.
Replicating data OpenSearch Service replicates your data to the cloned cluster.
Customer application validation The cloned cluster is ready. After preparation, test your application on the new version before deciding whether to cut over. The upgrade waits for you.
Traffic shift OpenSearch Service sends the share of read traffic that you chose to the cloned cluster.
Cut-over to new environment OpenSearch Service cuts over to the cloned cluster.
Complete upgrade OpenSearch Service finishes the upgrade.
Deleting older resources OpenSearch Service deletes the resources associated with the old environment.

While the cloned cluster waits for you, the ConfigChangeStatus is PendingUserInput. The response also includes TrafficSplit, which shows the share of read traffic that goes to the primary cluster and to the cloned cluster.

Validate the cloned cluster

While the upgrade waits, your domain keeps serving traffic from the primary cluster. Writes always go to the primary cluster, so you can validate reads only. Use any of the following methods.

Send a share of read traffic to the cloned cluster

Use UpdateClone with TrafficSplitOptions to send a percentage of read requests, from 0 to 100, to the cloned cluster. The percentage applies to search requests, such as _search, _msearch, _count, and _mget. Other requests, including all writes, go to the primary cluster. You can change the percentage as often as you like. Set it to 0 to send all reads back to the primary cluster.

Use OpenSearch Dashboards on the cloned cluster

If your domain runs OpenSearch, open the cloned cluster's OpenSearch Dashboards at https://my-domain-endpoint/_dashboards/clone. To run the same queries on the primary cluster, use the standard Dashboards URL, https://my-domain-endpoint/_dashboards.

The clone URL is available only until you cut over. After cutover, the cloned cluster takes the place of the primary cluster, and you use the standard Dashboards URL.

Compare CloudWatch metrics

The cloned cluster publishes metrics to the same CloudWatch namespace as your domain. Each metric name starts with Clone., for example Clone.CPUUtilization. Monitor these metrics in CloudWatch to compare how your workload performs on the primary cluster (the earlier version) and the cloned cluster (the target version) before you decide to cut over. For a complete list of metrics, see Clone-based upgrade metrics.

Cut over to the cloned cluster

When you're satisfied with the cloned cluster, cut over to it with UpdateClone and CutoverOptions set to ClonedCluster. OpenSearch Service then does the following:

  1. Checks the health of both clusters. If either cluster has a red status, OpenSearch Service rejects the cutover.

  2. Blocks writes to the primary cluster so that the cloned cluster can catch up. Reads stay available.

  3. Waits for the cloned cluster to catch up. If the cloned cluster doesn't catch up within 5 minutes, OpenSearch Service unblocks writes and the cutover fails.

  4. Moves traffic to the cloned cluster. Your domain endpoint doesn't change.

  5. Deletes the primary cluster immediately, or keeps it for the retention period you set.

Important

Writes are blocked until cutover completes. Cut over during a period of low write traffic.

Cancel the upgrade before cutover

You can cancel at any time before you cut over. Call UpdateClone with CutoverOptions set to PrimaryCluster. OpenSearch Service deletes the cloned cluster, and your domain stays as it was. A cluster with a red status doesn't block a cancel.

Retention period

The retention period is how long OpenSearch Service keeps the primary cluster after cutover so that you can roll back. If you don't set a retention period, OpenSearch Service deletes the primary cluster immediately after cutover, and the only way to return to the earlier version is to restore a snapshot on a new domain.

  • The maximum retention period is 336 hours (14 days). To request a longer period, contact Amazon Support.

  • You can set it when you start the upgrade, or in a separate call before cutover. The period counts from when the cutover completes.

  • After cutover, you can shorten or extend the period with RetentionInHours.

  • To end the period early, call UpdateClone with RetentionAction. Use RollbackToPrimaryCluster to return to the primary cluster and delete the cloned cluster. Use DeletePrimaryCluster to delete the primary cluster now.

Roll back

During the retention period, you can return to the primary cluster. Call UpdateClone with RetentionAction set to RollbackToPrimaryCluster. OpenSearch Service returns to the primary cluster and deletes the cloned cluster.

Warning

Data written after cutover will be lost

Rollback restores the retained old environment. Any data written to the domain after cutover will not be present unless you replay it manually.

Charges

For information about charges, see Charges for upgrades.

Troubleshooting

Issue Description
Cloned cluster creation fails The stage that failed shows in the DescribeDomainChangeProgress response. The primary cluster isn't affected.
Data copy fails If the copy to the cloned cluster pauses or fails, the Replicating data stage fails. The primary cluster isn't affected.
Cutover is rejected One of the clusters has a red status. Fix the red indexes, and try again.
Cutover fails after blocking writes The cloned cluster didn't catch up within 5 minutes. OpenSearch Service unblocks writes automatically. Try again when write traffic is lower.