Troubleshooting replication

The majority of objects replicate within 15 minutes. The time that it takes Amazon S3 to replicate an object depends on several factors, including the source and destination Region pair, and the size of the object. For large objects, replication can take up to several hours. For visibility into replication times, you can use S3 Replication Time Control (S3 RTC).

If the object that is being replicated is large, wait a while before checking to see whether it appears in the destination. You can also check the replication status of the source object. If the object replication status is PENDING, Amazon S3 has not completed the replication. If the object replication status is FAILED, check the replication configuration that's set on the source bucket. Additionally, to receive information about failures during replication, you can set up Amazon S3 Event Notifications replication to receive failure events. For more information, see Receiving replication failure events with Amazon S3 Event Notifications.

You can call the HeadObject API operation to check the replication status of an object. The HeadObject API operation returns the PENDING,COMPLETED, or FAILED replication status of an object. In a response to a HeadObject API call, the replication status is returned in the x-amz-replication-status element.

aws s3api head-object --bucket my-bucket --key index.html

After HeadObject returns the objects with a FAILED replication status, you can use S3 Batch Replication to replicate those failed objects. Alternatively, you can re-upload the failed objects to the source bucket, which will initiate replication for the new objects.

In the replication configuration on the source bucket, verify the following:

Verify that versioning has not been suspended on any bucket in the replication configuration. Both the source and destination buckets must have versioning enabled.

If a replication rule is set to Change object ownership to the destination bucket owner, then the Amazon Identity and Access Management (IAM) role that's used for replication must have the s3:ObjectOwnerOverrideToBucketOwner permission. This permission is granted on the resource (in this case, the destination bucket). For example, the following Resource statement shows how to grant this permission on the destination bucket:

{
  "Effect":"Allow",
  "Action":[
    "s3:ObjectOwnerOverrideToBucketOwner"
  ],
  "Resource":"arn:aws-cn:s3:::DestinationBucket/*"
}

If the destination bucket is owned by another account, the owner of the destination bucket must also grant the s3:ObjectOwnerOverrideToBucketOwner permission to the source bucket owner through the destination bucket policy. To use the following example bucket policy, replace the user input placeholders with your own information:

{
  "Version": "2012-10-17",
  "Id": "Policy1644945280205",
  "Statement": [
    {
      "Sid": "Stmt1644945277847",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws-cn:iam::123456789101:role/s3-replication-role"
      },
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateTags",
        "s3:ObjectOwnerOverrideToBucketOwner"
      ],
      "Resource": "arn:aws-cn:s3:::DestinationBucket/*"
    }
  ]
}

If you're setting the replication configuration in a cross-account scenario, where the source and destination buckets are owned by different Amazon Web Services accounts,the destination buckets can't be configured as Requester Pays buckets. For more information, see Using Requester Pays buckets for storage transfers and usage.

If a bucket's source objects are encrypted with an Amazon Key Management Service (Amazon KMS) key, then the replication rule must be configured to include Amazon KMS-encrypted objects. Make sure to select Replicate objects encrypted with Amazon KMS under your Encryption settings in the Amazon S3 console. Then, select an Amazon KMS key for encrypting destination objects.

To use an Amazon KMS key that belongs to the destination account to encrypt the destination objects, the destination account must grant the kms:GenerateDataKey and kms:Encrypt permissions to the replication role in the KMS key policy. To use the following example statement in your KMS key policy, replace the user input placeholders with your own information:

{    
    "Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws-cn:iam::123456789101:role/s3-replication-role"
    },
    "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
    "Resource": "*"
}

If you use an asterisk (*) for the Resource statement in the Amazon KMS key policy, the policy grants permission to use the KMS key to only the replication role. The policy doesn't allow the replication role to elevate its permissions.

By default, the KMS key policy grants the root user full permissions to the key. These permissions can be delegated to other users in the same account. Unless there are Deny statements in the source KMS key policy, using an IAM policy to grant the replication role permissions to the source KMS key is sufficient.

If either the source or destination KMS keys grant permissions based on the encryption context, confirm that Amazon S3 Bucket Keys are turned on for the buckets. If the buckets have S3 Bucket Keys turned on, the encryption context must be the bucket-level resource, like this:

"kms:EncryptionContext:arn:aws-cn:arn": [
     "arn:aws-cn:s3:::SOURCE_BUCKET_NAME"
     ]
"kms:EncryptionContext:arn:aws-cn:arn": [
     "arn:aws-cn:s3:::DESTINATION_BUCKET_NAME"
     ]

In addition to the permissions granted by the KMS key policy, the source account must add the following minimum permissions to the replication role's IAM policy:

{
    "Effect": "Allow",
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": [
        "SourceKmsKeyArn"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "kms:GenerateDataKey",
        "kms:Encrypt"
    ],
    "Resource": [
        "DestinationKmsKeyArn"
    ]
}

For more information about how to replicate objects that are encrypted with Amazon KMS, see Replicating encrypted objects.

If the destination bucket is owned by another Amazon Web Services account, verify that the bucket owner has a bucket policy on the destination bucket that allows the source bucket owner to replicate objects. For an example, see Configuring replication when source and destination buckets are owned by different accounts.

If your objects still aren't replicating after you've validated the permissions, check for any explicit Deny statements in the following locations:

If an object replica doesn't appear in the destination bucket, the following issues might have prevented replication:

Suppose that in the replication configuration, you add a rule to replicate a subset of objects that have a specific tag. In this case, you must assign the specific tag key and value at the time the object is created in order for Amazon S3 to replicate the object. If you first create an object and then add the tag to the existing object, Amazon S3 doesn't replicate the object.

Use Amazon S3 Event Notifications to notify you of instances when objects do not replicate to their destination Amazon Web Services Region. Amazon S3 event notifications are available through Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), or Amazon Lambda. For more information, see Receiving replication failure events with Amazon S3 Event Notifications.

You can also view replication failure reasons by using Amazon S3 Event Notifications. To review the list of failure reasons, see Amazon S3 replication failure reasons.

Batch operation status is failed with reason: The job report could not be written to your report bucket.

This error occurs if the IAM role that's used for the Batch Operations job is unable to put the completion report into the location that was specified when you created the job. To resolve this error, check that the IAM role has PutObject permissions for the bucket where you want to save the Batch Operations completion report. It's a best practice to deliver the report to a bucket different from the source bucket.

Batch operation is completed with failures and Total failed is not 0.

This error occurs if there are insufficient object permissions issues with the Batch Replication job that is running. If you're using a replication rule for your Batch Replication job, make sure that the IAM role used for replication has the proper permissions to access objects from either the source or destination bucket. You can also check the Batch Replication completion report to review the specific Amazon S3 replication failure reason.

Batch job ran successfully but the number of objects expected in destination bucket is not the same.

This error occurs when there's a mismatch between the objects listed in the manifest that's supplied in the Batch Replication job and the filters that you selected when you created the job. You might also receive this message when the objects in your source bucket don't match any replication rules and aren't included in the generated manifest.