Managing large Amazon SQS messages using Python and Amazon S3 - Amazon Simple Queue Service
PrerequisitesConfiguring message storage Managing large Amazon SQS messages with Extended Client Library for Python
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Managing large Amazon SQS messages using Python and Amazon S3

You can use the Amazon Simple Queue Service Extended Client Library for Python and Amazon Simple Storage Service to manage large Amazon SQS messages. This is especially useful for consuming large message payloads, from 256 KB and up to 2 GB. The library saves the message payload to an Amazon S3 bucket and sends a message containing a reference of the stored Amazon S3 object to an Amazon Amazon SQS queue.

You can use the Extended Client Library for Python to do the following:

  • Specify whether payloads are always stored in Amazon S3, or only stored in S3 when a payload size exceeds 256 KB

  • Send a message that references a single message object stored in an Amazon S3 bucket

  • Retrieve the corresponding payload object from an Amazon S3 bucket

  • Delete the corresponding payload object from an Amazon S3 bucket

Prerequisites

The following are the prerequisites for using the Amazon SQS Extended Client Library for Python:

  • An Amazon account with the necessary credentials. To create an Amazon account, navigate to the Amazon home page , and then choose Create an Amazon Account . Follow the instructions. For information about credentials, see Credentials.

  • An Amazon SDK: The example on this page uses Amazon Python SDK Boto3. To install and set up the SDK, see the Amazon SDK for Python documentation in the Amazon SDK for Python Developer Guide

  • Python 3.x (or later) and pip.

  • The Amazon SQS Extended Client Library for Python, available from PyPI

Note

You can use the Amazon SQS Extended Client Library for Python to manage Amazon SQS messages using Amazon S3 only with the Amazon SDK for Python. You can't do this with the Amazon CLI, the Amazon SQS console, the Amazon SQS HTTP API, or any of the other Amazon SDKs.

Configuring message storage

The Amazon SQS Extended Client makes uses the following message attributes to configure the Amazon S3 message storage options:

  • large_payload_support: The Amazon S3 bucket name to store large messages.

  • always_through_s3: If True, then all messages are stored in Amazon S3. If False, messages smaller than 256 KB will not be serialized to the s3 bucket. The default is False.

  • use_legacy_attribute: If True, all published messages use the Legacy reserved message attribute (SQSLargePayloadSize) instead of the current reserved message attribute (ExtendedPayloadSize).

Managing large Amazon SQS messages with Extended Client Library for Python

The following example creates an Amazon S3 bucket with a random name. It then creates an Amazon SQS queue named MyQueue and sends a message that is stored in an S3 bucket and is more than 256 KB to the queue. Finally, the code retrieves the message, returns information about it, and then deletes the message, the queue, and the bucket. 


import boto3
import sqs_extended_client

#Set the Amazon SQS extended client configuration with large payload.
sqs_extended_client = boto3.client("sqs", region_name="us-east-1")
sqs_extended_client.large_payload_support = "S3_BUCKET_NAME" 
sqs_extended_client.use_legacy_attribute = False


# Create an SQS message queue for this example. Then, extract the queue URL.
queue = sqs_extended_client.create_queue(
    QueueName = "MyQueue"
)
queue_url = sqs_extended_client.get_queue_url(
    QueueName = "MyQueue"
)['QueueUrl']


# Create the S3 bucket and allow message objects to be stored in the bucket. 
sqs_extended_client.s3_client.create_bucket(Bucket=sqs_extended_client.large_payload_support)

# Sending a large message
small_message = "s"
large_message = small_message * 300000 # Shall cross the limit of 256 KB

send_message_response = sqs_extended_client.send_message(
    QueueUrl=queue_url,
    MessageBody=large_message
)
assert send_message_response['ResponseMetadata']['HTTPStatusCode'] == 200

# Receiving the large message
receive_message_response = sqs_extended_client.receive_message(
    QueueUrl=queue_url,
    MessageAttributeNames=['All']
)
assert receive_message_response['Messages'][0]['Body'] == large_message
receipt_handle = receive_message_response['Messages'][0]['ReceiptHandle']

# Deleting the large message
# Set to True for deleting the payload from S3
sqs_extended_client.delete_payload_from_s3 = True 
delete_message_response = sqs_extended_client.delete_message(
    QueueUrl=queue_url,
    ReceiptHandle=receipt_handle
)

assert delete_message_response['ResponseMetadata']['HTTPStatusCode'] == 200

# Deleting the queue
delete_queue_response = sqs_extended_client.delete_queue(
    QueueUrl=queue_url
)

assert delete_queue_response['ResponseMetadata']['HTTPStatusCode'] == 200