AWS offers many great messaging services. Two of their most famous services are Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS). While both can be used in a pretty similar way they are completely different services.
This blog post will explain to you the similarities, and differences, and how to choose which service.
Finally, I will show you some example use cases and a common event-driven pattern called the fanout pattern that combines SNS and SQS.
Let's go 🚀
Amazon Simple Notification Service (SNS)
Amazon's SNS is a fully managed publish and subscribe service. A publisher sends a message to a topic and many consumers/subscribers are subscribed to this topic. The relationship is many to many. You can have many publishers and many subscribers to a topic.
SNS differentiates in their sending methods. It is either Application to Application (A2A) or Application to Person (A2P).
Application to Application (A2A) destinations are:
AWS Lambda
Amazon SQS
Amazon Kinesis Data Firehose
AWS Event Fork Pipelines
HTTP Endpoints
Application to Person (A2P) destinations are:
SMS
Email
In-App notification
AWS Chatbot
PagerDuty
SNS is super performant. Messages will be published within milliseconds.
A common pattern is a fan-out pattern that allows one to fan out one event to various subscribers within AWS. I will go more into detail about that pattern later on in the use case section.
SNS allows standard topics or FIFO topics. FIFO topics have a message ordering and standard topics don't. For FIFO topics there are much harder limits!
Amazon Simple Queue Service (SQS)
Amazon SQS is a fully managed distributed queueing service. SQS is poll-based and not push-based. Even, if it often seems like a push-based system, it is not. Amazon SQS is often used to decouple systems from each other and to enable asynchronous workloads.
The main pattern of SQS is to have producers send messages to a queue. The message remains in the queue for a defined time (by default 4 days, maximum 14 days). See our article on the retention period. Consumers can pick up the messages on their schedule by checking the queue if any new messages are available.
If a consumer works on a message the message will be deleted if it is successful. Otherwise, it can also be picked up by other consumers.
SQS offers many capabilities for retrying messages with its redrive policy. You can define several retries and a dead letter queue in case messages are failing. Dead letter queues (DLQ) are used to handle messages with errors. If messages can't be worked on they will be sent to a DLQ to inform the application developer about the issues and optionally save the message to replay it in the original queue.
SQS has a many-to-one relationship. You can send messages to a queue from many different producers but only one consumer can be defined. A consumer is another application, most often some compute instances such as Lambda, EC2, or Fargate.
There are two different types of queues: Standard queues and First-in, first-out (FIFO) queues. The latter one will keep the messages in order.
Fun fact: SQS is AWS's oldest service.
Differences
This article focuses on the differences between both services.
We understand that both services handle messages in some way. Both services enable better decoupling. The backend API and the execution of background logic are loosely coupled and not connected anymore.
But there are notable differences. Let's have a look at all of them.
Push and poll-based
The main difference lies in the foundation of the services. SQS is poll-based and SNS is a push-based service.
That means SNS is simply forwarding all messages to your subscribed consumers, whereas SQS saves the messages in a queue and waits till they get picked up. This is a notable difference in various aspects. The latency in SQS architectures for example is a bit higher since the polling still needs to be accounted for. The persistence and reliability on the other hand are with SQS much better since the message is properly saved for a short period.
Many-to-many vs. many-to-one (Number of Consumers)
A second difference is the type of relationship. Both services can receive messages from different producers. That means both services have a many-to-x relationship.
The main difference is that SNS can have a lot of subscribers whereas SQS can only have one consumer.
The current limit of SNS subscribers is 12,500,000 subscribers per topic. That is a lot! This means you can have many many consumers who will work on your message.
SQS on the other hand can only have one consumer. This consumer is working on the message and deletes the message afterward.
Types of Consumers
SNS sends their messages either to an application directly to a person, or both. That means it supports a variety of different consumer types.
SNS can be sent to many different destinations. These are:
Application to Application (A2A) destinations are:
AWS Lambda
Amazon SQS
Amazon Kinesis Data Firehose
AWS Event Fork Pipelines
HTTP Endpoints
Application to Person (A2P) destinations are:
SMS
Email
In-App notification
AWS Chatbot
PagerDuty
SQS messages on the other hand will typically be picked up using the SQS API. So every client that supports the AWS SDK can use it. Typically, messages in queues will be picked up from AWS Lambda because there is native integration with SQS and Lambda. But it is also possible to simply pick up and remove a message with the SQS API. You can also do it from your local PC 😉
Persistence
Messages in SQS will be saved for some duration. This is called the retention period. The retention period can be between 1 minute and 14 days and its default value is 4 days. If the message wasn't picked up within that timeframe the message will be removed automatically. Here is a more detailed article.
In SNS however, no persistence exists. There is no guarantee that the message will be delivered. If a consumer is not available the message won't be delivered.
This can make quite a difference in the reliability. If a consumer for example is not available in SNS the message will simply not be delivered. Or if a consumer doesn't end successfully the message is simply gone. SQS adds a lot of reliability to that. The fanout pattern can be used to combine both, but later more on that topic.
Reliability - Retries
SQS can add a Redrive Policy. This policy defines how many times a failed message should be retried before it will be moved to a Dead Letter Queue (DLQ). The DLQ handles failed messages. For example, you could save failed messages in a bucket and inform the developer about them.
SNS doesn't offer retries when the client fails. In case a consumer is not available or the consumer fails to work on the message (e.g. push notification won't come through) the message can't be repeated. This is due to the asynchronous nature of SNS.
Batching
SQS allows you to batch multiple messages together into one. You can define the parameter batch_size. The batch size can be a maximum of 10,000 records for standard queues and a maximum of 10 for FIFO queues.
SNS only works on one message at a time, so no batching is possible.
When to use what
When should I use SNS and when should I use SQS now?
Some general advice:
Use SNS if:
You have many subscribers
You need to send to consumers of the type SMS, Email, or app notifications
You want to use a fanout pattern to send a message to many subscribers at the same time (later more)
Use SQS if:
you only need one subscriber
persistence and error handling are really important (each message needs to be delivered)
you need to batch your requests
you want to simply decouple your application and enable async background processing
Use Cases
Let's have some example use cases.
SNS
CloudWatch Alarms:
An alarm will be triggered and you want to send messages to 10 different email addresses and SMS to some mobile phones. Persistence, batching, and retries are not important.
In-App Notification
A new user followed you on your new social media application. You want to inform the user about that.
SQS
Background Processing
Your application is synchronously doing all tasks and your user needs to wait till your API returns. By adding an SQS queue you can run background tasks and decouple the whole application.
Batch Processing
You need to work on a large number of messages at the same time. SQS can handle that by handling all messages in batch.
One example:
You host a startup show and you get a huge number of votes at the same time. You need to process these votes. It is not feasible that your API handles all of that and you don't want your users to wait the time it takes to process them.
You can use SQS for that. When a user votes your API sends a message to SQS. Your API returns 200 OK to the end-user and the end-user gets a super-fast response.
The actual business logic is decoupled. SQS and Lambda will take care of the rest. In such scenarios, many messages will be batched together and many lambda functions will be spawned. This is what scalability looks like.
SNS and SQS in Combination - Fanout pattern
In the article so far we've looked at SNS and SQS as two distinguished services. There is a common pattern out there called which combines both services.
It is called the fanout pattern.
The fanout pattern describes a scenario where messages published to SNS will be sent to several endpoints at the same time. With a pattern like that one message may trigger several executions. This allows for asynchronous processing.
For example:
A user uploads a picture to your application. You want to have several background processes kicking off when this happens. The following things should happen:
Start image recognition and create a headline
Process the image and create a thumbnail
Notify the users
All of these processes can run at the same time in the background.
Another example could be a social media network.
For each post published several actions should be taken. place:
SQS Queue 1: Translate post to different languages
SQS Queue 2: Transform post into audio
SQS Queue 3: Update user statistics (number of posts)
Email: Notify followers about new post
In-App: Notify followers about new post
Questions
There are a lot of questions I hear a lot when people start learning SNS and SQS. Let's have a look at some of the most common ones.
When should I use SNS vs. SQS?
SNS:
Multiple receivers
The receiver type is Email, SMS, or an In-App Notification
The message needs to be forwarded instantly
SQS:
Reliability is important
You have only one consumer
Complicated retry and error handling
Messages are picked up after some time and not instantly
What other alternatives are out there?
One alternative to mention is EventBridge. There are use cases where the choice for SNS or EventBridge overlaps. EventBridge allows you to send events to a special event bus and EventBridge forwards your event to services such as SNS, SQS, Lambda, etc.
One of the main differences is that EventBridge allows routing and filtering based on the whole event. There are also many different limits such as the 5 targets per event rule.
Another common message service is Kinesis. Kinesis is meant for processing and analyzing streaming data.
When to use Kinesis vs. SNS vs. SQS?
Kinesis:
Ongoing stream of data
Data needs to be processed or analyzed
Complex real-time requirements
SNS:
Multiple receivers
The receiver type is Email, SMS, or an In-App Notification
The message needs to be forwarded instantly
SQS:
Reliability is important
You have only one consumer
Complicated retry and error handling
Messages are picked up after some time and not instantly
Is SQS push or poll?
SQS is a poll service. Consumers poll for messages. Even if you use an EventSourceMapping with AWS Lambda. The lambda is still polling multiple times and receives a lot of empty responses.
This is where SQS can get quite expensive if you have many queues that get polled a lot.
Which targets are available for SQS?
SQS can be used with a lot of different targets. All systems that can call the AWS API can receive and remove messages from a queue.
SQS will often be used in combination with:
AWS Lambda
Amazon ECS
Amazon EC2
Amazon SNS
But you can use it with almost all systems that are out there.
Is SNS push or poll?
SNS is a push system. It won't persist the message at all but push it directly to its consumers.
Can I use SNS for Push Notifications?
Yes! You can use SNS for push notifications on iOS and Android for example.
Final Words
This article gave you an introduction to both AWS Services Amazon Simple Queue Service (SQS) and Amazon Simple Notification Service (SNS). These services build the basics of many distributed and decoupled applications out there. The service has existed for many many years and was one of the main pillars of AWS. Using these services to build reliable and performant applications is crucial.