SQS Implementation of Consumer Group Concept of Kafka

Both Kafka and SQS can be used as message queue to coordinate communication among micro services. Kafka uses the concept of consumer group while AWS SQS requires creating multiple queues with message fan-out. While we use Kafka in a lot of data ingestion and pipelines, the major advantage of AWS SQS is that it is serverless – you don’t need to pay on per hour basis and rather pay only when you invoke SQS APIs. Please read how Alden Investment Group benefited from Serverless Architecture.

The architecture requires two processes:

1. Message Fan-out
2. Message processing

Message Fan-out

A message may need to be sent to multiple consumer groups. For example when an online e-commerce payment concludes, the financial partner sends payment successful or failure message. The message needs to be communicated to billing services to create an invoice and it needs to be sent to dispatch service to prepare for delivery.

In these cases, the message is first sent to AWS SNS and SNS further redirects to one SQS queue per consumer group (one for billing service and one for dispatch service). This architecture is scalable because one may begin with just a couple of SQS receivers and may add more when more services require the same message.

Message processing

As the volume of transaction grows, one may need to add a number of consumers per service (concept of consumer group). Each of these consumers should consume a message only once and all of these consumers should consume all the messages in the queue per unit time. SQS achieves this requirement by providing visibility to messages.

A consumer reads one or a number of messages (AWS recommends batch size of 10). As soon as a message is read, the message becomes invisible in the queue to all consumers. Other consumers may continue to pick messages which are visible.

The consumer then processes the read messages.

1. If consumer determines that it has successfully processed a message then it should delete that message from SQS.

2. If consumer determines that it does not want to process that message or if consumer fails while processing this message then it shouldn’t take any further action.

3. A message read by one consumer and not deleted by that consumer, becomes visible to all consumers after Visibility timeout. Now any of the consumers may pick this re-visible message and process it.

4. What if consumers continue to fail while processing a message: When a message has been received by consumers more than maxReceiveCount setting of SQS then the message is moved to Dead Letter Queue and it is not processed by SQS consumers.

Back to you

Do you use Apache Kafka, SQS or something else for your message publish-subscribe needs? We would love to hear from you.

About VisionFirst Technologies Pvt. Ltd.

We are “AWS Registered Consulting Partner”. Our tech stack includes Public Cloud, Hybrid Cloud, Serverless Architecture, Serverless Databases, Data pipeline to process 100+ million messages per day, Highly Available, Highly Scalable and Highly Fault Tolerant micro-service Applications, precision UX mobile apps, Web applications, Bank-grade secured applications (Cyber Security), IOT, Machine Learning/Deep Learning and Big Data Technologies.

Please contact us to know how we may help you.