Using failover in an Amazon Aurora global database

An Aurora global database provides more comprehensive failover capabilities than the failover provided by a default Aurora DB cluster. By using an Aurora global database, you can plan for and recover from disaster fairly quickly. Recovery from disaster is typically measured using values for RTO and RPO.

With an Aurora global database, there are two different approaches to failover depending on the scenario.

Recovering an Amazon Aurora global database from an unplanned outage

On very rare occasions, your Aurora global database might experience an unexpected outage in its primary Amazon Web Services Region. If this happens, your primary Aurora DB cluster and its writer node aren't available, and the replication between the primary cluster and the secondaries ceases. To minimize both downtime (RTO) and data loss (RPO), you can work quickly to perform a cross-Region failover and reconstruct your Aurora global database.

Make sure that application writes are sent to the correct Aurora DB cluster before, during, and after making these changes. Doing this avoids data inconsistencies among the DB clusters in the Aurora global database (split-brain issues).

If you reconfigured in response to an outage in an Amazon Web Services Region, you can make that Amazon Web Services Region the primary again after the outage is resolved. To do so, you add the old Amazon Web Services Region to your new global database, and then use the managed planned failover process to switch its role. Your Aurora global database must use a version of Aurora PostgreSQL or Aurora MySQL that supports managed planned failovers. For more information, see Performing managed planned failovers for Amazon Aurora global databases.

Performing managed planned failovers for Amazon Aurora global databases

By using managed planned failovers, you can relocate the primary cluster of your Aurora global database to a different Amazon Web Services Region on a routine basis. This approach is intended for controlled environments, such as operational maintenance and other planned operational procedures.

As an example, say a financial institution headquartered in New York has branch offices located in San Francisco, the UK, and Europe. The organization's core business applications use an Aurora global database. Its primary cluster runs in the US East (Ohio) Region. It has secondary clusters running in the US West (N. California) Region, Europe (London) Region, and the Europe (Frankfurt) Region. Every quarter, it relocates the primary cluster from the (current) primary Amazon Web Services Region to the secondary Region designated for that rotation.

During a managed planned failover, your primary cluster is failed over to your choice of secondary Region while your Aurora global database's existing replication topology is maintained. Before the managed planned failover process begins, Aurora global database synchronizes all secondary clusters with its primary cluster. After ensuring that all clusters are synchronized, the managed planned failover begins. The DB cluster in the primary Region becomes read-only. The chosen secondary cluster promotes one of its read-only nodes to full writer status, thus allowing the cluster to assume the role of primary cluster. Because all secondary clusters were synchronized with the primary at the beginning of the process, the new primary continues operations for the Aurora global database without losing any data. Your database is unavailable for a short time while the primary and selected secondary clusters are assuming their new roles.

To optimize application availability, we recommend that you do the following before using this feature:

During a managed planned failover, the chosen secondary DB cluster is promoted to its new role as primary. However, it doesn't inherit the various configuration options of the primary DB cluster. A mismatch in configuration can lead to performance issues, workload incompatibilities, and other anomalous behavior. To avoid such issues, we recommend that you resolve differences between your Aurora global database clusters for the following:

When the failover process completes, the promoted Aurora DB cluster can handle write operations for the Aurora global database. Make sure to change the endpoint for your application to use the new endpoint. If you accepted the provided names when you created the Aurora global database, you can change the endpoint by removing the -ro from the promoted cluster's endpoint string in your application.

For example, the secondary cluster's endpoint my-global.cluster-ro-aaaaaabbbbbb.us-west-1.rds.amazonaws.com becomes my-global.cluster-aaaaaabbbbbb.us-west-1.rds.amazonaws.com when that cluster is promoted to primary.

If you are using RDS Proxy, make sure to redirect your application's write operations to the appropriate read/write endpoint of the proxy that's associated with the new primary cluster. This proxy endpoint might be the default endpoint or a custom read/write endpoint. For more information see How RDS Proxy endpoints work with global databases.

You can fail over your Aurora global database using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

To start the failover process on your Aurora global database

1. Sign in to the Amazon Web Services Management Console and open the Amazon RDS console at https://console.amazonaws.cn/rds/.

2. Choose Databases and find the Aurora global database you want to fail over.

3. Choose Fail over global database in the Actions menu. The failover process doesn't begin until after you choose the failover target in the next step. At this point, the failover is pending.

   

   1. Choose the secondary Aurora DB cluster that you want to promote to primary. The secondary DB cluster must be available. If you have more than one secondary DB cluster, you can compare the lag amount for all secondaries and choose the one with the smallest amount of lag.

      

   2. Choose Fail over global database to confirm your choice of secondary DB cluster and begin the failover process.

      Tip

      The failover process can take some time to complete. You can cancel once the process is underway, but it can take some time to return your Aurora global database to its original configuration.

      

      The Status column of the Databases list shows the state of each Aurora DB instance and Aurora DB cluster during the failover process.

      

      The status bar at the top of the Console displays progress and provides a Cancel failover option. If you choose Cancel failover, you're given the option to proceed with the failover or to cancel the failover process.

      

      Choose Cancel failover if you want to cancel the failover.

   3. Choose Close to continue failing over and dismiss the prompt.

When the failover completes, you can see the Aurora DB clusters and their current state in the Databases list, as shown following.

To fail over an Aurora global database

Use the failover-global-cluster CLI command to fail over your Aurora global database. With the command, pass values for the following parameters.

• --region – Specify the Amazon Web Services Region where the primary DB cluster of the Aurora global database is running.

• --global-cluster-identifier – Specify the name of your Aurora global database.

• --target-db-cluster-identifier – Specify the Amazon Resource Name (ARN) of the Aurora DB cluster that you want to promote to be the primary for the Aurora global database.

For Linux, macOS, or Unix:

aws rds --region aws-Region \
   failover-global-cluster --global-cluster-identifier global_database_id \
  --target-db-cluster-identifier ARN-of-secondary-to-promote

For Windows:

aws rds --region aws-Region ^
   failover-global-cluster --global-cluster-identifier global_database_id ^
  --target-db-cluster-identifier ARN-of-secondary-to-promote

To fail over an Aurora global database, run the FailoverGlobalCluster API operation.

Managing RPOs for Aurora PostgreSQL–based global databases

With an Aurora PostgreSQL–based global database, you can manage the recovery point objective (RPO) for your Aurora global database by using PostgreSQL's rds.global_db_rpo parameter. RPO represents maximum amount of data that can be lost in the event of an outage.

When you set an RPO for your Aurora PostgreSQL–based global database, Aurora monitors the RPO lag time of all secondary clusters to make sure that at least one secondary cluster stays within the target RPO window. RPO lag time is another time-based metric.

The RPO is used when your database resumes operations in a new Amazon Web Services Region after a failover. Aurora evaluates RPO and RPO lag times to commit (or block) transactions on the primary as follows:

In other words, if all secondary clusters are behind the target RPO, Aurora pauses transactions on the primary cluster until at least one of the secondary clusters catches up. Paused transactions are resumed and committed as soon as the lag time of at least one secondary DB cluster becomes less than the RPO. The result is that no transactions can commit until the RPO is met.

If you set this parameter as outlined in the following, you can then also monitor the metrics that it generates. You can do so by using psql or another tool to query the Aurora global database's primary DB cluster and obtain detailed information about your Aurora PostgreSQL–based global database's operations. To learn how, see Monitoring Aurora PostgreSQL-based Aurora global databases.

Setting the recovery point objective

The rds.global_db_rpo parameter controls the RPO setting for a PostgreSQL database. This parameter is supported by Aurora PostgreSQL. Valid values for rds.global_db_rpo range from 20 seconds to 2,147,483,647 seconds (68 years). Choose a realistic value to meet your business need and use case. For example, you might want to allow up to 10 minutes for your RPO, in which case you set the value to 600.

You can set this value for your Aurora PostgreSQL–based global database by using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

aws rds modify-db-cluster-parameter-group \
    --db-cluster-parameter-group-name my_custom_global_parameter_group \
    --parameters "ParameterName=rds.global_db_rpo,ParameterValue=600,ApplyMethod=immediate"

aws rds modify-db-cluster-parameter-group ^
    --db-cluster-parameter-group-name my_custom_global_parameter_group ^
    --parameters "ParameterName=rds.global_db_rpo,ParameterValue=600,ApplyMethod=immediate"

Viewing the recovery point objective

The recovery point objective (RPO) of a global database is stored in the rds.global_db_rpo parameter for each DB cluster. You can connect to the endpoint for the secondary cluster you want to view and use psql to query the instance for this value.

db-name=>show rds.global_db_rpo;

If this parameter isn't set, the query returns the following:

rds.global_db_rpo
-------------------
 -1
(1 row)

This next response is from a secondary DB cluster that has 1 minute RPO setting.

rds.global_db_rpo
-------------------
 60
(1 row)

You can also use the CLI to get values for find out if rds.global_db_rpo is active on any of the Aurora DB clusters by using the CLI to get values of all user parameters for the cluster.

For Linux, macOS, or Unix:

aws rds describe-db-cluster-parameters \
 --db-cluster-parameter-group-name lab-test-apg-global \
 --source user

For Windows:

aws rds describe-db-cluster-parameters ^
 --db-cluster-parameter-group-name lab-test-apg-global *
 --source user

The command returns output similar to the following for all user parameters. that aren't default-engine or system DB cluster parameters.

{
    "Parameters": [
        {
            "ParameterName": "rds.global_db_rpo",
            "ParameterValue": "60",
            "Description": "(s) Recovery point objective threshold, in seconds, that blocks user commits when it is violated.",
            "Source": "user",
            "ApplyType": "dynamic",
            "DataType": "integer",
            "AllowedValues": "20-2147483647",
            "IsModifiable": true,
            "ApplyMethod": "immediate",
            "SupportedEngineModes": [
                "provisioned"
            ]
        }
    ]
}

To learn more about viewing parameters of the cluster parameter group, see Viewing parameter values for a DB cluster parameter group.

Disabling the recovery point objective

To disable the RPO, reset the rds.global_db_rpo parameter. You can reset parameters using the Amazon Web Services Management Console, the Amazon CLI, or the RDS API.

aws rds reset-db-cluster-parameter-group \
    --db-cluster-parameter-group-name global_db_cluster_parameter_group \
    --parameters "ParameterName=rds.global_db_rpo,ApplyMethod=immediate"

aws rds reset-db-cluster-parameter-group ^
    --db-cluster-parameter-group-name global_db_cluster_parameter_group ^
    --parameters "ParameterName=rds.global_db_rpo,ApplyMethod=immediate"