Using encryption with the Amazon S3 event crawler - Amazon Glue
Enabling encryption on SQS onlyEnabling encryption on both SQS and Amazon S3FAQ
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Using encryption with the Amazon S3 event crawler

This section describes using encryption on SQS only or on both SQS and Amazon S3.

Enabling encryption on SQS only

Amazon SQS provides encryption in-transit by default. To add optional Server-Side Encryption (SSE) to your queue you can attach a customer master key (CMK) in the edit panel. This means that SQS encrypts all customer data at-rest on SQS servers.

Create a Customer Master Key (CMK)

  1. Choose Key Management Service (KMS) > Customer Managed Keys > Create key.

  2. Follow the steps to add your own alias and description.

  3. Add the respective IAM roles you would like to be able to use this key.

  4. In the key policy, add another statement to the "Statement" list so that your custom key policy gives the Amazon SNS sufficient key usage permissions.

     "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "sns.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
 ]
Enable Server-Side Encryption (SSE) on your queue

  1. Choose Amazon SQS > Queues > sqs_queue_name > Encryption tab.

  2. Choose Edit, and scroll down to the Encryption drop down.

  3. Select Enabled to add SSE.

  4. Select the CMK you created earlier, and not the default key with the name alias/aws/sqs.

    After adding this, your Encryption tab is updated with the key you added.

Note

Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default message retention period is 4 days. To avoid missing events change the SQS MessageRetentionPeriod to the maximum of 14 days.

Enabling encryption on both SQS and Amazon S3

Enable Server-Side Encryption (SSE) on SQS

  1. Follow the steps in Enabling encryption on SQS only.

  2. In the last step of the CMK setup, give Amazon S3 sufficient key usage permissions.

    Paste the following in to the "Statement" list:

     "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
 ]
Enable Server-Side Encryption (SSE) on your Amazon S3 bucket

  1. Follow the steps in Enabling encryption on SQS only.

  2. Do one of the following:

    • To enable SSE for your entire S3 bucket, navigate to the Properties tab in your target bucket.

      Here you can enable SSE and choose the encryption type you would like to use. Amazon S3 provides an encryption key that Amazon S3 creates, manages, and uses for you, or you can choose a key from KMS as well.

    • To enable SSE on a specific folder, click the checkbox beside your target folder and choose Edit server-side encryption under the Actions drop down.

FAQ

Why aren't messages that I publish to my Amazon SNS topic getting delivered to my subscribed Amazon SQS queue that has server-side encryption (SSE) enabled?

Double check that your Amazon SQS queue is using:

  1. A customer master key (CMK) that is customer managed. Not the default one provided by SQS.

  2. Your CMK from (1) includes a custom key policythat gives the Amazon SNS sufficient key usage permissions.

For more information, see this article in the knowledge center.

I’ve subscribed to email notifications, but I don’t receive any email updates when I edit my Amazon S3 bucket.

Make sure that you have confirmed your email address by clicking the "Confirm Subscription" link in your email. You can verify the status of your confirmation by checking the Subscriptions table under your SNS topic.

Choose Amazon SNS > Topics > sns_topic_name > Subscriptions table.

If you followed our prerequisite script, you will find that the sns_topic_name is equal to your sqs_queue_name. It should look similar to the following:

Only some of the folders I added are showing up in my table after enabling server-side encryption on my SQS queue. Why am I missing some parquets?

If the Amazon S3 bucket changes were made before enabling SSE on your SQS queue, they may not be picked up by the crawler. To ensure that you have crawled all the updates to your S3 bucket, run the crawler again in listing mode ("Crawl All Folders"). Another option is to start fresh by creating a new crawler with S3 events enabled.