Amazon Simple Queue Service (also known as SQS) is a fully managed queue service that helps application decoupling. SQS allows to send, store and consume messages at any volume. There are no restrictions on the number of queues that you can create on the number of messages published on each SQS queue. The overall data flow for SQS is as below:

  1. The publisher publishes the message to the SQS queue asynchronously.
  2. The SQS service stores the messages.
  3. The consumer or group of consumers then reads the messages from SQS, processes them. The consumer is responsible for deleting the message from the SQS queue after processing it successfully.
  4. In this way, the producer doesn’t have to wait for the consumer process to finish the message processing, hence decoupling the application. 

You can refer to the SQS notes below if you are planning to appear for various AWS certification exams like:

In this article, we will talk about the following points in detail.

AWS SQS – Functionality

  • There is no restriction on the number of SQS queues you can create.
  • Each message size can be between 1KB to 256KB.
  • If your message size is greater than 256KB, you can use Amazon SQS extended client. It writes the data on AWS S3 and sends the S3 file metadata information on the SQS queue.
  • SQS supports batching for send, read and delete operations. It allows processing ten messages at once, saving the cost of API requests.
  • SQS supports long polling. The consumer with long polling can wait up to 20 seconds for a new message to come up on SQS. 
  • SQS can store the data for a maximum of 14 days.
  • SQS supports the dead letter queue, which is another queue. If consumers cannot process the messages, SQS sends the messages to such a dead letter queue.
  • SQS supports data encryption.
  • SQS also supports message visibility, ensuring multiple consumers consuming the data from the same SQS queue get different sets of the message to process.

Types of AWS SQS Queues

AWS SQS supports two types of queues: FIFO (First In, First Out) & Standard. You should use a FIFO queue if message ordering is essential. Let’s discuss both queue types and detail.

SQS FIFO (First In, First Out)

  • SQS FIFO support processing 300 messages per second. With batching enabled, it supports 3000 messages per second as each batch can have a maximum of 10 messages.
  • SQS FIFO has the option of allowing high throughput, which provides processing 3000 messages per second or 30000 messages per second with batching enabled.
  • SQS FIFO ensures exactly-once delivery semantics.
  • SQS FIFO provides that it is preserving the message ordering.

SQS Standard

  • Standard SQS queue support unlimited throughput for reading, write and delete operations. You can add more publishers and consumers to get more throughput.
  • The message ordering is not guaranteed to be the same on the consumer side. 
  • Standard SQS queues support at least once delivery semantics, so you will occasionally see the message delivered twice.

AWS SQS – Queue naming convention

  • SQS queue name can have a maximum of 80 characters and care case-sensitive.
  • SQS queue names can have alphanumeric characters, a hyphen, and underscores.
  • The FIFO SQS name must end with suffix .fifo

AWS SQS – Visibility Timeout

  • AWS SQS allows multiple consumers to process the data. It can lead to a situation where numerous consumers are processing the same set of records. SQS supports a message visibility timeout to avoid such scenarios.
  • The message read by one consumer will be unavailable for other consumers for the period of message visibility timeout.
  • The consumer must process the message within visibility timeout and delete it from the SQS queue to avoid another consumer processing the same message.
  • By default, the message visibility timeout is 30 seconds. The minimum value for SQS message visibility timeout is 0 seconds, and the maximum value is 12 hours.

AWS SQS – Message Retention period

  • AWS SQS stores unprocessed data for a maximum of 14 days. You can delete the unprocessed messages earlier than 14 days. This duration is known as Message Retention Duration.
  • Any message on the SQS queue older than the message retention period gets deleted by SQS.
  • The message retention timeout for Deal letter queues must be greater than the message retention period of the original SQS queue.

AWS SQS – Delivery Delay

  • SQS messages will not be visible to any consumer till the delivery delay is over.
  • It helps when the consumer requires extra time to process previous messages.
  • Changing the delivery delay time for Standard SQS will not affect existing messages in the queue.
  • Changing the delivery delay for the FIFO SQS queue will affect existing messages in the SQS queue.

AWS SQS – Maximum message size

  • SQS message size can be between 1BYTE to 256KB.
  • You can set the maximum SQS message size.
  •  You can use SQS extended client library, which allows sending a much larger payload. It copied the message to S3 and sends file metadata as an SQS message.
  • SQS extended client library supports the load up to 2GB.

AWS SQS – Receive message wait time

  • SQS client polls the queue for checking the availability of new messages.
  • Depending on how frequently clients can poll the SQS queue, there are two types of polling: short polling and long polling.
  • If you set SQS’s received message wait time to 0, it becomes short polling where the client continuously keeps polling for new messages. 
  • Short polling can increase the cost, and it constantly polls the SQS queue.
  • If you set SQS’s received message wait time between 1 – 20 seconds, it becomes short polling where the consumer waits for a particular duration for a new message to arrive before sending a new request.
  • Long polling helps to reduce the cost.

AWS SQS – Content-based deduplication

  • Content-based deduplication applies only to SQS FIFO queues.
  • It ensures that if the same message is received more than once, it sends the message only once for processing to the consumer.
  • If you enable this SQS feature, sending deduplication id becomes optional.
  • It helps in use cases where you must process the messages exactly once.

AWS SQS – High throughput FIFO

  • It applies only to SQS FIFO queues.
  • By default, FIFO queue support processing 300 messages per second (or 3000 messages per second with batching enabled).
  • If you need more throughput, you can enable this option. It allows you to get a throughput of 3000 messages per second (30000 messages per second with batching)
  • If you enable this option, SQS will set message deduplication scope to “message group.”
  • If you enable this option, SQS will set the FIFO throughput to “per message group ID.”

AWS SQS – Deduplication scope

  • It applied only to SQS FIFO queues.
  • It supports two types of deduplication scope:
    • Queue  where SQS will deduplicate message based on complete queue
    • Message Group where SQS will deduplicate message messaged on message ID. 

AWS SQS – FIFO throughput limit

  • It applies only to SQS FIFO queues.
  • You will use the SQS throughput limit to either one of the below:
    • Queue
    • Per message group ID.

AWS SQS – Access Policy

  • SQS Access policy defines who can access the SQS queue.
  • A policy statement is a JSON object containing keys:
    • Effect
      • The effect could be Allow or Deny.
    • Principal
    • Action
      • It contains the list of all actions which will be effective for the principal. e.g., SNS: Publish, SNS: Subscribe, and so on 
    • Resource
      • Resource for which you are defining the policy
  • The default policy statement that allows only the queue owner to send, receive or delete looks like below:
{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "123456789012"
      },
      "Action": [
        "SQS:*"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:"
    }
  ]
}

AWS SQS – Encryption at rest

  • SQS supports encryption at rest.
  • Once enabled, SQS will encrypt the message body before storing it on the SQS queue.
  • SQS will not encrypt queue metadata, message metadata, and per-queue metrics.
  • It is server-side encryption as SQS takes care of encryption and decrypting the data.
  • SQS will decrypt the message just before delivering it to SQS consumers.
  • You will have to provide custom CMK to SQS so that SQS can encrypt and decrypt the data.
  • When SQS moves the message from SQS to the dead letter queue, it stays encrypted.
  • If you apply the encryption to the existing SQS queue, SQS will not encrypt already existing messages.

AWS SQS – Dead-letter Queue

  • SQS Dead letter queue ensures the messages that consumers can’t process are not being reprocessed infinitely. Instead, SQS will move such messages if you have configured a dead letter queue.
  • SQS dead letter queue is another queue where SQS will move messages which SQS Consumers process cant.
  • You must configure FIFO SQS with FIFO dead letter queue.
  • The maximum receive value decides when to send a message to a dead letter. Let’s say you configured the maximum receive value to 3; SQS will send a message to the dead letter once the consumer fails to process it three times.

AWS SQS – Tags

  • SQS Tag is key-value pair that you can assign to each SQS topic.
  • Each tag consists of a tag and an optional value. 
  • YOu can use the tags to filter AWS resources based on tags.
  • Tags are also helpful in tracking the resource cost.

Conclusion

In this article, we discussed all AWS SQS concepts in depth.