Skip to Content

Does WhatsApp use message queue?

WhatsApp is one of the most popular messaging apps in the world, with over 2 billion active users. Given its massive scale, managing the flow of messages between users is a significant technical challenge. This raises the question – does WhatsApp use message queues to help process the high volume of messages sent through its platform?

What are message queues?

A message queue is a form of asynchronous service-to-service communication used in server applications. Message queues allow different parts of an application to communicate and process operations asynchronously, improving performance, reliability and scalability.

Here is how message queues work:

  • An application publishes a message to a queue.
  • The messaging system keeps the message until it is consumed.
  • Other applications subscribe to the queues and consume messages.
  • The subscriber handles the message then sends an acknowledgement when processing is complete.

Message queues decouple components so producers and consumers can work at different paces. They also act as a buffer in times of peak loads. This prevents bottlenecks and improve resiliency against outages.

Benefits of using message queues

Here are some key benefits of using message queues:

  • Asynchronous communication – Senders and receivers don’t need to interact with the message queue at the same time.
  • Durability – Messages persist in the queue until received and processed.
  • Scalability – Queues can handle huge volumes of messages and spikes in traffic.
  • Redundancy – Messages can be duplicated across multiple queues to prevent data loss.
  • Performance – Queues increase overall application performance by decoupling processes.

Does WhatsApp use message queues?

Given the advantages of message queues, it is highly likely that WhatsApp leverages them as part of its architecture. There are several clues that suggest WhatsApp relies on message queues:

  • WhatsApp needs to handle an extremely high volume of messages sent each day across its large user base. A messaging system helps absorb these huge spikes in traffic.
  • Users do not need to be online at the same time to receive messages. This indicates asynchronous communication compatible with messaging queues.
  • WhatsApp continues to successfully deliver messages even when users temporarily lose connectivity. Queues provide the durability for this.
  • WhatsApp claim they have designed their systems to be ‘highly available, scalable, and reliable’. This aligns with the benefits of message queues.

While WhatsApp has not publicly confirmed the exact details of their architecture, various sources indicate the usage of message queues. According to an ex-WhatsApp engineer, they use RabbitMQ to manage queues internally. RabbitMQ is a popular open source message queuing system. WhatsApp is also reported to use Kafka, another distributed queuing system created by LinkedIn.

WhatsApp’s technical architecture

WhatsApp operates a globally distributed system to handle its billions of users. Here are some key components of their architecture:

Application servers

WhatsApp’s core application logic is hosted on Erlang application servers. Erlang is known for high availability, distribution and fault tolerance. This provides the reliability needed for WhatsApp’s messaging operations.

Messenger queues

As mentioned above, WhatsApp likely implements dedicated messenger queues for asynchronous communication between their apps, servers and services. These queues help absorb the massive throughput of daily messages.

Media storage

Media like photos and videos are likely stored on distributed file storage systems. This allows fast access to media when users request it.

Databases

WhatsApp utilizes a customized Erlang database along with Cassandra, a popular NoSQL database. Data is partitioned across these databases for performance and scalability.

CDN

WhatsApp uses a content delivery network (CDN) like Fastly to cache media and deliver it quickly to users. The CDN edge servers provide low latency access to frequently requested media.

Client apps

Native mobile and web apps allow users to access WhatsApp’s messaging services. These are built using cross-platform frameworks like React Native.

How messages flow through WhatsApp

Here is a simplified overview of how messages may flow through WhatsApp’s queue-based architecture:

  1. A user sends a message via the WhatsApp client app.
  2. The app publishes the message to a send queue.
  3. Application servers consume messages from this queue.
  4. Servers process and validate the message then publish it to a delivery queue.
  5. The message waits in the delivery queue until received by the recipient’s device.
  6. The recipient’s app consumes the message and displays it to the user.
  7. An acknowledgment is sent back to confirm successful delivery.

Media messages likely follow a similar path, with files being stored and retrieved from WhatsApp’s distributed file servers. WhatsApp may use separate queues for different types of messages and traffic.

Managing potential issues

Operating queue-based systems at massive scale comes with potential pitfalls that WhatsApp likely mitigates through careful monitoring and engineering, including:

Data loss

Messages could be lost before fully processed. WhatsApp implements redundancy by replicating queues and using message delivery acknowledgments.

Duplicate delivery

Errors could cause messages to be delivered more than once. WhatsApp provides mechanisms to detect and discard duplicates.

Ordering

Messages could be consumed out of order. WhatsApp may use ordered queues or sequence numbers to enforce proper ordering.

Poison messages

A malformed message could crash receivers. WhatsApp likely implements poison message handling to detect and discard them.

Scalability

Spikes in traffic could overwhelm queues. WhatsApp scales horizontally by distributing queues across servers.

Conclusion

In conclusion, while WhatsApp has not confirmed the exact details publicly, there are strong indications that they rely on message queues and related asynchronous messaging patterns to handle the enormous traffic on their platform. Message queues provide vital benefits like scalability, redundancy and performance for a massively distributed architecture. Careful design around messaging semantics, reliability and ordering prevents issues like data loss and duplication. Overall, message queues seem essential for WhatsApp to efficiently and reliably process the billions of daily messages sent by their users worldwide.