Using switchover or failover in an Amazon Aurora global database

An Aurora global database provides more business continuity and disaster recovery (BCDR) protection than the standard high availability provided by an Aurora DB cluster in a single Amazon Web Services Region. By using an Aurora global database, you can plan for and recover from true Regional disasters or complete service-level outages quickly. Recovery from disaster is typically driven by the following two business objectives:

Switching over or failing over an Aurora global database involves promoting a DB cluster in one of your global database's secondary Regions to be the primary DB cluster. The term "regional outage" is often used to describe a variety of failure scenarios. A worst case scenario could be a wide-spread outage from a catastrophic event that affects hundreds of square miles. However, most outages are much more localized, affecting only a small subset of cloud services or customer systems. Consider the full scope of the outage to make sure cross-Region failover is the proper solution and to choose the appropriate failover method for the situation. Whether you should use the failover or switchover approach depends on the specific outage scenario:

Recovering an Amazon Aurora global database from an unplanned outage

On very rare occasions, your Aurora global database might experience an unexpected outage in its primary Amazon Web Services Region. If this happens, your primary Aurora DB cluster and its writer node aren't available, and the replication between the primary and secondary DB clusters stops. To minimize both downtime (RTO) and data loss (RPO), you can work quickly to perform a cross-Region failover.

There are two methods for failing over in a disaster recovery situation:

Performing managed failovers for Aurora global databases

This approach is intended for business continuity in the event of a true Regional disaster or complete service-level outage.

During a managed failover, your primary cluster is failed over to your choice of secondary Region while your Aurora global database's existing replication topology is maintained. The chosen secondary cluster promotes one of its read-only nodes to full writer status. This step allows the cluster to assume the role of primary cluster. Your database is unavailable for a short time while this cluster is assuming its new role. Data that wasn't replicated from the old primary to the chosen secondary cluster is missing when this secondary becomes the new primary.

To minimize data loss, we recommend that you do the following before using this feature:

During a managed failover, the chosen secondary DB cluster is promoted to its new role as primary. However, it doesn't inherit the various configuration options of the primary DB cluster. A mismatch in configuration can lead to performance issues, workload incompatibilities, and other anomalous behavior. To avoid such issues, we recommend that you resolve differences between your Aurora global database clusters for the following:

Typically, the chosen secondary cluster assumes the primary role within a few minutes. As soon as the new primary Region's writer node is available, you can connect your applications to it and resume your workloads. After Aurora promotes the new primary cluster, it automatically rebuilds all additional secondary Region clusters.

Because Aurora global databases use asynchronous replication, the replication lag in each secondary Region can vary. Aurora rebuilds these secondary Regions to have the exact same point-in-time data as the new primary Region cluster. The duration of the complete rebuilding task can take a few minutes to several hours, depending on the size of the storage volume and the distance between the Regions. When the secondary Region clusters finish rebuilding from the new primary Region, they become available for read access.

As soon as the new primary writer is promoted and available, the new primary Region's cluster can handle read and write operations for the Aurora global database. Make sure to change the endpoint for your application to use the new endpoint. If you accepted the provided names when you created the Aurora global database, you can change the endpoint by removing the -ro from the promoted cluster's endpoint string in your application.

For example, the secondary cluster's endpoint my-global.cluster-ro-aaaaaabbbbbb.us-west-1.rds.amazonaws.com becomes my-global.cluster-aaaaaabbbbbb.us-west-1.rds.amazonaws.com when that cluster is promoted to primary.

If you are using RDS Proxy, make sure to redirect your application's write operations to the appropriate read/write endpoint of the proxy that's associated with the new primary cluster. This proxy endpoint might be the default endpoint or a custom read/write endpoint. For more information see How RDS Proxy endpoints work with global databases.

To restore the global database cluster's original topology, Aurora monitors the availability of the old primary Region. As soon as that Region is healthy and available again, Aurora automatically adds it back to the global cluster as a secondary Region. Before creating the new storage volume in the old primary Region, Aurora tries to take a snapshot of the old storage volume at the point of failure. It does this so that you can use it to recover any of the missing data. If this operation is successful, Aurora places this snapshot named "rds:unplanned-global-failover-name-of-old-primary-DB-cluster-timestamp" in the snapshot section of the Amazon Web Services Management Console. You can also see this snapshot listed in the information returned by the DescribeDBClusterSnapshots API operation.

After the original topology is restored, you can fail back your global database to the original primary Region by performing a switchover operation when it makes the most sense for your business and workload. To do so, follow the steps in Performing switchovers for Amazon Aurora global databases.

You can fail over your Aurora global database using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

To perform the managed failover on your Aurora global database

1. Sign in to the Amazon Web Services Management Console and open the Amazon RDS console at https://console.amazonaws.cn/rds/.

2. Choose Databases and find the Aurora global database you want to fail over.

3. Choose Switch over or fail over global database from the Actions menu.

   

4. Choose Failover (allow data loss).

   

5. For New primary cluster, choose an active cluster in one of your secondary Amazon Web Services Regions to be the new primary cluster.

6. Enter confirm, and then choose Confirm.

When the failover completes, you can see the Aurora DB clusters and their current state in the Databases list, as shown in the following image.

To perform the managed failover on an Aurora global database

Use the failover-global-cluster CLI command to fail over your Aurora global database. With the command, pass values for the following parameters.

• --region – Specify the Amazon Web Services Region where the secondary DB cluster that you want to be the new primary for the Aurora global database is running.

• --global-cluster-identifier – Specify the name of your Aurora global database.

• --target-db-cluster-identifier – Specify the Amazon Resource Name (ARN) of the Aurora DB cluster that you want to promote to be the new primary for the Aurora global database.

• --allow-data-loss – Explicitly make this a failover operation instead of a switchover operation. A failover operation can result in some data loss if the asynchronous replication components haven't completed sending all replicated data to the secondary Region.

For Linux, macOS, or Unix:

aws rds --region region_of_selected_secondary \
   failover-global-cluster --global-cluster-identifier global_database_id \
   --target-db-cluster-identifier arn_of_secondary_to_promote \
   --allow-data-loss

For Windows:

aws rds --region region_of_selected_secondary ^
   failover-global-cluster --global-cluster-identifier global_database_id ^
   --target-db-cluster-identifier arn_of_secondary_to_promote ^
   --allow-data-loss

To fail over an Aurora global database, run the FailoverGlobalCluster API operation.

Performing manual failovers for Aurora global databases

In some scenarios, you might not be able to use the managed failover process. One example is if your primary and secondary DB clusters aren't running compatible engine versions. In this case, you can follow this manual process to fail over your global database to your target secondary Region.

Make sure that application writes are sent to the correct Aurora DB cluster before, during, and after making these changes. Doing this avoids data inconsistencies among the DB clusters in the Aurora global database (split-brain issues).

If you reconfigured in response to an outage in an Amazon Web Services Region, you can make that Amazon Web Services Region the primary again after the outage is resolved. To do so, you add the old Amazon Web Services Region to your new global database, and then use the switchover process to switch its role. Your Aurora global database must use a version of Aurora PostgreSQL or Aurora MySQL that supports switchovers. For more information, see Performing switchovers for Amazon Aurora global databases.

Performing switchovers for Amazon Aurora global databases

By using switchovers, you can change the Region of your primary cluster on a routine basis. This approach is intended for controlled scenarios, such as operational maintenance and other planned operational procedures.

There are three common use cases for using switchovers.

During a switchover, Aurora switches over your primary cluster to your chosen secondary Region while it maintains your global database's existing replication topology. Before it starts the switchover process, Aurora waits for all secondary Region clusters to be fully synchronized with the primary Region cluster. Then, the DB cluster in the primary Region becomes read-only and the chosen secondary cluster promotes one of its read-only nodes to full writer status. Promoting this node to a writer allows that secondary cluster to assume the role of primary cluster. Because all secondary clusters were synchronized with the primary at the beginning of the process, the new primary continues operations for the Aurora global database without losing any data. Your database is unavailable for a short time while the primary and selected secondary clusters are assuming their new roles.

To optimize application availability, we recommend that you do the following before using this feature:

During a switchover, the chosen secondary DB cluster is promoted to its new role as primary. However, it doesn't inherit the various configuration options of the primary DB cluster. A mismatch in configuration can lead to performance issues, workload incompatibilities, and other anomalous behavior. To avoid such issues, we recommend that you resolve differences between your Aurora global database clusters for the following:

When the switchover process completes, the promoted Aurora DB cluster can handle write operations for the Aurora global database. Make sure to change the endpoint for your application to use the new endpoint. If you accepted the provided names when you created the Aurora global database, you can change the endpoint by removing the -ro from the promoted cluster's endpoint string in your application.

For example, the secondary cluster's endpoint my-global.cluster-ro-aaaaaabbbbbb.us-west-1.rds.amazonaws.com becomes my-global.cluster-aaaaaabbbbbb.us-west-1.rds.amazonaws.com when that cluster is promoted to primary.

If you are using RDS Proxy, make sure to redirect your application's write operations to the appropriate read/write endpoint of the proxy that's associated with the new primary cluster. This proxy endpoint might be the default endpoint or a custom read/write endpoint. For more information see How RDS Proxy endpoints work with global databases.

You can switch over your Aurora global database using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

To perform the switchover on your Aurora global database

1. Sign in to the Amazon Web Services Management Console and open the Amazon RDS console at https://console.amazonaws.cn/rds/.

2. Choose Databases and find the Aurora global database you want to switch over.

3. Choose Switch over or fail over global database from the Actions menu.

   

4. Choose Switchover.

   

5. For New primary cluster, choose an active cluster in one of your secondary Amazon Web Services Regions to be the new primary cluster.

6. Choose Confirm.

When the switchover completes, you can see the Aurora DB clusters and their current roles in the Databases list, as shown in the following image.

To perform the switchover on an Aurora global database

Use the switchover-global-cluster CLI command to switch over your Aurora global database. With the command, pass values for the following parameters.

• --region – Specify the Amazon Web Services Region where the primary DB cluster of the Aurora global database is running.

• --global-cluster-identifier – Specify the name of your Aurora global database.

• --target-db-cluster-identifier – Specify the Amazon Resource Name (ARN) of the Aurora DB cluster that you want to promote to be the primary for the Aurora global database.

For Linux, macOS, or Unix:

aws rds --region region_of_primary \
   switchover-global-cluster --global-cluster-identifier global_database_id \
  --target-db-cluster-identifier arn_of_secondary_to_promote

For Windows:

aws rds --region region_of_primary ^
   switchover-global-cluster --global-cluster-identifier global_database_id ^
  --target-db-cluster-identifier arn_of_secondary_to_promote

To switch over an Aurora global database, run the SwitchoverGlobalCluster API operation.

Managing RPOs for Aurora PostgreSQL–based global databases

With an Aurora PostgreSQL–based global database, you can manage the recovery point objective (RPO) for your Aurora global database by using the rds.global_db_rpo parameter. RPO represents the maximum amount of data that can be lost in the event of an outage.

When you set an RPO for your Aurora PostgreSQL–based global database, Aurora monitors the RPO lag time of all secondary clusters to make sure that at least one secondary cluster stays within the target RPO window. RPO lag time is another time-based metric.

The RPO is used when your database resumes operations in a new Amazon Web Services Region after a failover. Aurora evaluates RPO and RPO lag times to commit (or block) transactions on the primary as follows:

In other words, if all secondary clusters are behind the target RPO, Aurora pauses transactions on the primary cluster until at least one of the secondary clusters catches up. Paused transactions are resumed and committed as soon as the lag time of at least one secondary DB cluster becomes less than the RPO. The result is that no transactions can commit until the RPO is met.

The rds.global_db_rpo parameter is dynamic. If you decide that you don't want all write transactions to stall until the lag decreases sufficiently, you can reset it quickly. In this case, Aurora recognizes and implements the change after a short delay.

If you set this parameter as outlined in the following, you can then also monitor the metrics that it generates. You can do so by using psql or another tool to query the Aurora global database's primary DB cluster and obtain detailed information about your Aurora PostgreSQL–based global database's operations. To learn how, see Monitoring Aurora PostgreSQL-based global databases.

Setting the recovery point objective

The rds.global_db_rpo parameter controls the RPO setting for a PostgreSQL database. This parameter is supported by Aurora PostgreSQL. Valid values for rds.global_db_rpo range from 20 seconds to 2,147,483,647 seconds (68 years). Choose a realistic value to meet your business need and use case. For example, you might want to allow up to 10 minutes for your RPO, in which case you set the value to 600.

You can set this value for your Aurora PostgreSQL–based global database by using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

aws rds modify-db-cluster-parameter-group \
    --db-cluster-parameter-group-name my_custom_global_parameter_group \
    --parameters "ParameterName=rds.global_db_rpo,ParameterValue=600,ApplyMethod=immediate"

aws rds modify-db-cluster-parameter-group ^
    --db-cluster-parameter-group-name my_custom_global_parameter_group ^
    --parameters "ParameterName=rds.global_db_rpo,ParameterValue=600,ApplyMethod=immediate"

Viewing the recovery point objective

The recovery point objective (RPO) of a global database is stored in the rds.global_db_rpo parameter for each DB cluster. You can connect to the endpoint for the secondary cluster you want to view and use psql to query the instance for this value.

db-name=>show rds.global_db_rpo;

If this parameter isn't set, the query returns the following:

rds.global_db_rpo
-------------------
 -1
(1 row)

This next response is from a secondary DB cluster that has 1 minute RPO setting.

rds.global_db_rpo
-------------------
 60
(1 row)

You can also use the CLI to get values for find out if rds.global_db_rpo is active on any of the Aurora DB clusters by using the CLI to get values of all user parameters for the cluster.

For Linux, macOS, or Unix:

aws rds describe-db-cluster-parameters \
 --db-cluster-parameter-group-name lab-test-apg-global \
 --source user

For Windows:

aws rds describe-db-cluster-parameters ^
 --db-cluster-parameter-group-name lab-test-apg-global *
 --source user

The command returns output similar to the following for all user parameters. that aren't default-engine or system DB cluster parameters.

{
    "Parameters": [
        {
            "ParameterName": "rds.global_db_rpo",
            "ParameterValue": "60",
            "Description": "(s) Recovery point objective threshold, in seconds, that blocks user commits when it is violated.",
            "Source": "user",
            "ApplyType": "dynamic",
            "DataType": "integer",
            "AllowedValues": "20-2147483647",
            "IsModifiable": true,
            "ApplyMethod": "immediate",
            "SupportedEngineModes": [
                "provisioned"
            ]
        }
    ]
}

To learn more about viewing parameters of the cluster parameter group, see Viewing parameter values for a DB cluster parameter group.

Disabling the recovery point objective

To disable the RPO, reset the rds.global_db_rpo parameter. You can reset parameters using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

aws rds reset-db-cluster-parameter-group \
    --db-cluster-parameter-group-name global_db_cluster_parameter_group \
    --parameters "ParameterName=rds.global_db_rpo,ApplyMethod=immediate"

aws rds reset-db-cluster-parameter-group ^
    --db-cluster-parameter-group-name global_db_cluster_parameter_group ^
    --parameters "ParameterName=rds.global_db_rpo,ApplyMethod=immediate"