FIFO queue delivery logic in Amazon SQS - Amazon Simple Queue Service
Sending messagesReceiving messagesRetrying multiple timesAdditional notes on FIFO behaviorExamples for better understanding
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

FIFO queue delivery logic in Amazon SQS

The following concepts clarify how Amazon SQS FIFO queues handle the sending and receiving of messages, particularly when dealing with message ordering and message group IDs.

Sending messages

Amazon SQS FIFO queues preserve message order using unique deduplication IDs and message group IDs. This topic highlights the importance of message group IDs for maintaining strict ordering within groups and highlights best practices for ensuring reliable, ordered message delivery across multiple producers.

  1. Order preservation

    • When multiple messages are sent in succession to a FIFO queue with unique message deduplication IDs, Amazon SQS stores them and acknowledges their transmission. These messages are then received and processed in the exact order they were transmitted.

  2. Message group ID

    • In FIFO queues, messages are ordered based on their message group ID. If multiple producers or threads send messages with the same message group ID, Amazon SQS ensures they are stored and processed in the order they arrive.

    • Best practice: To guarantee strict message order across multiple producers, assign a unique message group ID for all messages from each producer.

  3. Per-group ordering

    • FIFO queue logic applies on a per message group ID basis:

      • Each message group ID represents a distinct, ordered group of messages.

      • Within a message group ID, all messages are sent and received in strict order.

      • Messages with different message group IDs may arrive or be processed out of order relative to one another.

    • Requirement - You must associate a message group ID with each message. If a message is sent without a group ID, the action fails.

    • Single group scenario - If you require all messages to be processed in strict order, use the same message group ID for every message.

Receiving messages

Amazon SQS FIFO queues handle message retrieval, including batch processing, FIFO order guarantees, and limitations on requesting specific message group IDs. This topic explains how Amazon SQS retrieves messages within and across message group IDs while maintaining strict ordering and visibility rules.

  1. Batch retrieval

    • When receiving messages from a FIFO queue with multiple message group IDs, Amazon SQS:

      • Attempts to return as many messages as possible with the same message group ID in a single call.

      • Allows other consumers to process messages from different message group IDs concurrently.

    • Important clarification

      • You may receive multiple messages from the same message group ID in one batch (up to 10 messages in a single call using the MaxNumberOfMessages parameter).

      • However, you can't receive additional messages from the same message group ID in subsequent requests until:

        • The currently received messages are deleted, or

        • They become visible again (for example, after the visibility timeout expires).

  2. FIFO order guarantee

    • Messages retrieved in a batch retain their FIFO order within the group.

    • If fewer than 10 messages are available for the same message group ID, Amazon SQS may include messages from other message group IDs in the same batch, but each group retains FIFO order.

  3. Consumer limitations

    • You cannot explicitly request to receive messages from a specific message group ID.

Retrying multiple times

Producers and consumers can safely retry failed actions in Amazon SQS FIFO queues without disrupting message order or introducing duplicates. This topic highlights how deduplication IDs and visibility timeouts ensure message integrity during retries.

  1. Producer retries

    • If a SendMessage action fails, the producer can retry sending the message multiple times with the same message deduplication ID.

    • As long as the producer receives at least one acknowledgment before the deduplication interval expires, retries:

      • Do not introduce duplicate messages.

      • Do not disrupt message order.

  2. Consumer retries

    • If a ReceiveMessage action fails, the consumer can retry as many times as necessary using the same receive request attempt ID.

    • As long as the consumer receives at least one acknowledgment before the visibility timeout expires, retries:

      • Do not disrupt message order.

Additional notes on FIFO behavior

Learn about handling visibility timeouts, enabling parallel processing with multiple message group IDs, and ensuring strict sequential processing in single-group scenarios.

  1. Handling visibility timeout

    • When a message is retrieved but not deleted, it remains invisible until the visibility timeout expires.

    • No additional messages from the same message group ID are returned until the first message is deleted or becomes visible again.

  2. Concurrency and parallel processing

    • FIFO queues allow parallel processing of messages across different message group IDs.

    • To maximize concurrency, design your system with multiple message group IDs for independent workflows.

  3. Single group scenarios

    • For strict sequential processing of all messages in a FIFO queue, use a single message group ID for all messages in the queue.

Examples for better understanding

The following are practical scenarios illustrating FIFO queue behavior in Amazon SQS.

  1. Scenario 1: Single group ID

    • A producer sends five messages with the same message group ID Group A.

    • A consumer receives these messages in FIFO order. Until the consumer deletes these messages or the visibility timeout expires, no additional messages from Group A are received.

  2. Scenario 2: Multiple group IDs

    • A producer sends five messages to Group A and 5 to Group B.

    • Consumer 1 processes messages from Group A, while Consumer 2 processes messages from Group B. This enables parallel processing with strict ordering maintained within each group.

  3. Scenario 3: Batch retrieval

    • A producer sends seven messages to Group A and three to Group B.

    • A single consumer retrieves up to 10 messages. If the queue allows, it may return:

      • Seven messages from Group A and three from Group B (or fewer if fewer messages are available from a single group).