Fault tolerance

Checks if there are Amazon DocumentDB clusters configured as Single-AZ.

Running Amazon DocumentDB workloads in a Single-AZ architecture is not sufficient for highly critical workloads and it can take up to 10 minutes to recover from a component failure. Customers should deploy replica instances in additional availability zones to ensure availability during maintenance, instance failures, component failures, or availability zone failures.

c15vnddn2x

Yellow: Amazon DocumentDB cluster has instances in less than three availability zones.

Green: Amazon DocumentDB cluster has instances in three availability zones.

If your application requires high availability, modify your DB instance to enable Multi-AZ using replica instances. See Amazon DocumentDB High Availability and Replication

Understanding Amazon DocumentDB Cluster Fault Tolerance

Regions and Availability Zones

Checks the age of the snapshots for your Amazon EBS volumes (either available or in-use). Failures can occur even if Amazon EBS volumes are replicated. Snapshots are persisted toAmazon S3 for durable storage and point-in-time recovery.

H7IgTzjTYb

Create weekly or monthly snapshots of your volumes. For more information, see Creating an Amazon EBS Snapshot.

To automate the creation of EBS snapshots, you can consider using Amazon Backup or Amazon Data Lifecycle Manager.

Amazon Elastic Block Store (Amazon EBS)

Amazon EBS Snapshots

Amazon Backup

Amazon Data Lifecycle Manager

Checks for Amazon ECS task definitions configured with the AmazonLogs logging driver in blocking mode. A driver configured in the blocking mode risks system availability.

c1dvkm4z6b

Yellow: The awslogs driver logging configuration parameter mode is set to blocking or missing. A missing mode parameter indicates a default blocking configurations.

Green: Amazon ECS task definition is not using the awslogs driver or the awslogs driver is configured in non-blocking mode.

To mitigate the availability risk, consider changing the task definition AmazonLogs driver configuration from blocking to non-blocking. With non-blocking mode, you will have to set a value for the max-buffer-size parameter. For more information and guidance on configuration parameters, see . See Preventing log loss with non-blocking mode in the AmazonLogs container log driver

Using the Amazon logs log driver

Choosing container logging options to avoid backpressure

Preventing log loss with non-blocking mode in the AmazonLogs container log driver

Checks for ElastiCache clusters that deploy in a single Availability Zone (AZ). This check alerts you if Multi-AZ is inactive in a cluster.

Deployments in multiple AZs enhance ElastiCache cluster availability by asynchronously replicating to read-only replicas in a different AZ. When planned cluster maintenance occurs, or a primary node is unavailable, ElastiCache automatically promotes a replica to primary. This failover allows cluster write operations to resume, and doesn't require an administrator to intervene.

ECHdfsQ402

Create at least one replica per shard, in an AZ that is different than the primary.

For more information, see Minimizing downtime in ElastiCache (Redis OSS) with Multi-AZ.

Checks for MemoryDB clusters that deploy in a single Availability Zone (AZ). This check alerts you if Multi-AZ is inactive in a cluster.

Deployments in multiple AZs enhance MemoryDB cluster availability by asynchronously replicating to read-only replicas in a different AZ. When planned cluster maintenance occurs, or a primary node is unavailable, MemoryDB automatically promotes a replica to primary. This failover allows cluster write operations to resume, and doesn't require an administrator to intervene.

MDBdfsQ401

Create at least one replica per shard, in an AZ that is different than the primary.

For more information, see Minimizing downtime in MemoryDB with Multi-AZ.

Checks the number of Availability Zones (AZs) for your Amazon MSK provisioned cluster. The Amazon MSK cluster is formed of several brokers that work together and distribute the data and load. Production might be interrupted during maintenance or broker issues in a 2-AZ cluster.

90046ff5b5

To increase availability of the cluster, you can create another cluster in a 3 AZs setup. Then migrate the existing cluster to the new cluster that you created. You can use Amazon MSK replication for this migration.

Amazon MSK high availability

Amazon MSK migration

Checks for automated backups of Amazon RDS DB instances.

By default, backups are enabled with a retention period of one day. Backups reduce the risk of unexpected data loss and allow for point-in-time recovery.

opQPADkZvH

Red: A DB instance has the backup retention period set to 0 days.

Set the retention period for the automated DB instance backup to 1 to 35 days as appropriate to the requirements of your application. See Working With Automated Backups.

Getting Started with Amazon RDS

Checks the logging configuration of Amazon Simple Storage Service (Amazon S3) buckets.

When server access logging is enabled, detailed access logs are delivered hourly to a bucket that you choose. An access log record contains details about each request, such as the request type, the resources specified in the request, and the time and date the request was processed. By default, bucket logging is not enabled. You should enable logging if you want to perform security audits or learn more about users and usage patterns.

When logging is initially enabled, the configuration is automatically validated. However, future modifications can result in logging failures. This check examines explicit Amazon S3 bucket permissions, but it does not examine associated bucket policies that might override the bucket permissions.

BueAdJ7NrP

Enable bucket logging for most buckets. See Enabling Logging Using the Console and Enabling Logging Programmatically.

If the target bucket permissions do not include the root account and you want Trusted Advisor to check the logging status, add the root account as a grantee. See Editing Bucket Permissions.

If the target bucket does not exist, select an existing bucket as a target or create a new one and select it. See Managing Bucket Logging.

If the target and source have different owners, change the target bucket to one that has the same owner as the source bucket. See Managing Bucket Logging.

If the log deliverer does not have write permissions for the target (write not enabled), grant Upload/Delete permissions to the Log Delivery group. See Editing Bucket Permissions.

Examines the health check configuration for Auto Scaling groups.

If Elastic Load Balancing is being used for an Auto Scaling group, the recommended configuration is to enable an Elastic Load Balancing health check. If an Elastic Load Balancing health check is not used, Auto Scaling can only act upon the health of the Amazon Elastic Compute Cloud (Amazon EC2) instance. Auto Scaling will not act on the application running on the instance.

CLOG40CDO8

If the Auto Scaling group has an associated load balancer, but the Elastic Load Balancing health check is not enabled, see Add an Elastic Load Balancing Health Check to your Auto Scaling Group.

If the Elastic Load Balancing health check is enabled, but no load balancer is associated with the Auto Scaling group, see Set Up an Auto-Scaled and Load-Balanced Application.

Amazon EC2 Auto Scaling User Guide

Checks the availability of resources associated with launch configurations and your Auto Scaling groups.

Auto Scaling groups that point to unavailable resources cannot launch new Amazon Elastic Compute Cloud (Amazon EC2) instances. When properly configured, Auto Scaling causes the number of Amazon EC2 instances to increase seamlessly during demand spikes, and decrease automatically during demand lulls. Auto Scaling groups and launch configurations that point to unavailable resources do not operate as intended.

8CNsSllI5v

If the load balancer has been deleted, either create a new load balancer or target group then associate it to the Auto Scaling group, or create a new Auto Scaling group without the load balancer. For information about creating a new Auto Scaling group with a new load balancer, see Set Up an Auto-Scaled and Load-Balanced Application. For information about creating a new Auto Scaling group without a load balancer, see Create Auto Scaling Group in Getting Started With Auto Scaling Using the Console.

If the AMI has been deleted, create a new launch template or launch template version using a valid AMI and associate it with an Auto Scaling group. See Create Launch Configuration in Getting Started With Auto Scaling Using the Console.

Checks the resilience of the Amazon Direct Connect used to connect your on-premises to each Direct Connect gateway or virtual private gateway.

This check alerts you if any Direct Connect gateway or virtual private gateway isn't configured with virtual interfaces across at least two distinct Direct Connect locations. Lack of location resiliency can result in unexpected downtime during maintenance, a fiber cut, a device failure, or a complete location failure.

c1dfpnchv2

Red: The Direct Connect gateway or virtual private gateway is configured with one or more virtual interfaces on a single Direct Connect device.

Yellow: The Direct Connect gateway or virtual private gateway is configured with virtual interfaces across multiple Direct Connect devices in a single Direct Connect location.

Green: The Direct Connect gateway or virtual private gateway is configured with virtual interfaces across two or more distinct Direct Connect locations.

To build Direct Connect location resiliency, you can configure the Direct Connect gateway or virtual private gateway to connect to at least two distinct Direct Connect locations. For more information, see Amazon Direct Connect Resiliency Recommendation.

Amazon Direct Connect Resiliency Recommendations

Amazon Direct Connect Failover Test

Checks for Outposts Racks balance. This evaluates if a customers Outposts instances are deployed across multiple Outposts Racks or to a single Outpost Rack. A single Outposts rack creates a single point of failure for issues that involve a single Rack (for example, environmental failures). These scenarios can be mitigated by deploying outposts across multiple Racks.

c243hjzrhn

If you are running production workloads on Amazon Outposts, then its a best practice to use the following resilient architecture. A single Amazon Outposts rack creates a single point of failure. Consider adding a second Amazon Outposts rack to that location with enough capacity for a failover event, and then distribute workloads across racks.

Failure mode 4: Racks or data centers

Checks for Classic load balancers that do not have connection draining enabled.

When connection draining is not enabled and you deregister an Amazon EC2 instance from a Classic load balancer, the Classic load balancer stops routing traffic to that instance and closes the connection. When connection draining is enabled, the Classic load balancer stops sending new requests to the deregistered instance but keeps the connection open to serve active requests.

7qGXsKIUw

Enable connection draining for the Classic load balancer. For more information, see Connection Draining and Enable or Disable Connection Draining for Your Load Balancer.

Elastic Load Balancing Concepts

Checks the target groups’ target distribution across Availability Zones (AZs) for Application Load Balancer (ALB), Network Load Balancer (NLB), and Gateway Load Balancer (GWLB).

This check doesn’t includes load balancers that are configured with a single AZ and where the difference in number of targets between the most and least populated AZ’s is equal to or lesser than 1.

b92b83d667

For better resilience, make sure that your targets groups have same number of targets across AZs.

Target groups for your Application Load Balancers

Register targets with your Application Load Balancer target group

Checks your load balancer configuration.

To help increase the level of fault tolerance in Amazon Elastic Compute Cloud (Amazon EC2) when using Elastic Load Balancing , we recommend running an equal number of instances across multiple Availability Zones in a Region. A load balancer that is configured accrues charges, so this is a cost-optimization check as well.

iqdCTZKCUp

Ensure that your load balancer points to active and healthy instances in at least two Availability Zones. For more information, see Add Availability Zone.

If your load balancer is configured for an Availability Zone with no healthy instances, or if there is an imbalance of instances across the Availability Zones, determine if all the Availability Zones are necessary. Omit any unnecessary Availability Zones and ensure there is a balanced distribution of instances across the remaining Availability Zones. For more information, see Remove Availability Zone.

Checks if your Network Firewalls are configured to use more than one Availability Zone (AZ) for firewall endpoints.

An AZ is a distinct location that’s insulated from failures in other zones. If the Network Firewall endpoint is deployed in only 1 AZ, then it can be a single point of failure and can impair workloads from other AZs using the Network Firewall for traffic inspection. It’s a best practice to configure your Network Firewalls in multiple AZs in the same Region to mprove your workload availability.

c2vlfg0gqd

Make sure that your Network Firewall is configured with at least two AZs for production workloads.

VPC subnet configuration forAmazon Network Firewall

Creating a firewall

Availability Zone

Amazon Well-Architected Tool - Deploy the workload to multiple locations

Appliance in a shared services VPC