Checking object integrity for data at rest in Amazon S3

Each batch operations job progresses through different job statuses. You can also enable Amazon CloudTrail events in the S3 console to receive alerts on any job state changes. For active jobs, you can review the running job and completion rate on the Job details page.

You can use the create-job command to create a new batch operations job, and to provide the list of objects. Then, specify the checksum algorithm and checksum type, and the destination bucket where you want to save the Compute checksum report. The following example creates an S3 Batch Operations Compute checksum job by using an S3 generated manifest for the Amazon Web Services account 111122223333.

To use this command, replace the user input placeholders with your own information:

aws s3control create-job \
    --account-id 111122223333 \
    --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::my-manifest-bucket/manifest.csv","ETag":"e0e8bfc50e0f0c5d5a1a5f0e0e8bfc50"}}' \
    --manifest-generator '{
        "S3JobManifestGenerator": {
          "ExpectedBucketOwner": "111122223333",
          "SourceBucket": "arn:aws:s3:::amzn-s3-demo-source-bucket",
          "EnableManifestOutput": true,
          "ManifestOutputLocation": {
            "ExpectedManifestBucketOwner": "111122223333",
            "Bucket": "arn:aws:s3:::amzn-s3-demo-manifest-bucket",
            "ManifestPrefix": "prefix",
            "ManifestFormat": "S3InventoryReport_CSV_20211130"
          },
          "Filter": {
            "CreatedAfter": "2023-09-01",
            "CreatedBefore": "2023-10-01",
            "KeyNameConstraint": {
              "MatchAnyPrefix": [
                "prefix"
              ],
              "MatchAnySuffix": [
                "suffix"
              ]
            },
            "ObjectSizeGreaterThanBytes": 100,
            "ObjectSizeLessThanBytes": 200,
            "MatchAnyStorageClass": [
              "STANDARD",
              "STANDARD_IA"
            ]
          }
        }
      }' \
    --operation '{"S3ComputeObjectChecksum":{"ChecksumAlgorithm":"CRC64NVME","ChecksumType":"FULL_OBJECT"}}' \
    --report '{"Bucket":"arn:aws:s3:::my-report-bucket","Format":"Report_CSV_20180820","Enabled":true,"Prefix":"batch-op-reports/","ReportScope":"AllTasks","ExpectedBucketOwner":"111122223333"}' \
    --priority 10 \
    --role-arn arn:aws:iam::123456789012:role/S3BatchJobRole \
    --client-request-token 6e023a7e-4820-4654-8c81-7247361aeb73 \
    --description "Compute object checksums" \
    --region us-west-2

After you submit the Compute checksum job, you receive the job ID as a response and it appears on the S3 Batch Operations list page. Amazon S3 processes the list of objects and calculates checksums for each object. After the job finishes, S3 provides a consolidated Compute checksum report at the specified destination.

To monitor the progress of your Compute checksum job, use the describe-job command. This command checks the status of the specified batch operations job. To use this command, replace the user input placeholders with your own information.

For example:

aws s3control describe-job --account-id 111122223333 --job-id 1234567890abcdef0

To obtain a list of all Active and Complete batch operations jobs, see Listing jobs or list-jobs in the Amazon CLI Command Reference.

You can send REST requests to verify object integrity with Compute checksum using CreateJob. You can monitor the progress of Compute checksum requests by sending REST requests to the DescribeJob API operation. Each batch operations job progresses through the following statuses:

The API response notifies you of the current job state.