WhatsApp is one of the most popular messaging apps in the world, with over 2 billion active users. Under the hood, WhatsApp relies on a robust database architecture to store and manage the incredible amount of data generated by its billions of users.
Overview of WhatsApp’s Database Structure
WhatsApp utilizes a NoSQL database built on top of Erlang, a programming language known for its real-time, low-latency performance. The WhatsApp database is structured as a large key-value store, with each chat message assigned a unique ID that serves as the key. Additional metadata like sender ID, recipient ID, timestamp, etc. are stored as fields in the value portion.
This schemaless NoSQL structure allows WhatsApp to efficiently store and retrieve the billions of messages sent daily across its platform. It also enables easier horizontal scaling of the database as WhatsApp’s user base continues to grow.
Some key properties of WhatsApp’s database architecture include:
- Built on Erlang for real-time performance
- NoSQL key-value structured database
- Schemaless design for flexibility
- Messages stored as unique ID keys with metadata in values
- Designed for horizontal scaling
Message Storage
The core of WhatsApp’s infrastructure is message storage and retrieval. Whenever a user sends a message, it must be efficiently stored and replicated across WhatsApp’s servers. The message also needs to be immediately delivered to the recipient when they come online.
To facilitate this, each message is assigned a unique 64-bit integer ID when created. This message ID serves as the key in the key-value database structure. The value portion stores all the relevant metadata such as:
- Sender ID
- Recipient ID
- Timestamp
- Message text, media, etc.
- Message status (sent, delivered, read, etc.)
WhatsApp leverages Erlang’s real-time capabilities to instantly replicate newly created messages across nodes. The message ID key allows WhatsApp to efficiently fetch the message contents from the database when required.
To save storage space, media attachments like photos or videos are stored separately from the message database. The media files are assigned a separate ID, and this media ID is referenced from the message contents to link them together.
Message Delivery
When User A sends a message to User B, the message must be delivered to User B’s device when they come online. To enable this instant message delivery, WhatsApp maintains the current device status of every user.
User devices periodically send live ping messages to WhatsApp’s servers to indicate they are online. When User A sends a message to User B, WhatsApp checks User B’s device status. If online, the message is immediately pushed to their device. Otherwise, it is queued for delivery until User B comes online.
This architecture allows WhatsApp to support real-time messaging delivery worldwide, even for recipients who temporarily lose connectivity.
Contact and Group Storage
In addition to chat messages, WhatsApp needs to store and manage contacts and groups. This includes the contact details, group metadata, group participants, etc.
Similar to messages, all contacts and groups are assigned a unique ID when created. These IDs become the keys to fetch their details from the database as needed.
Contact details like phone numbers are stored in a separate contacts table that is indexed for fast lookups. This avoids duplicating contact details across multiple chats.
Group data is structured as nested objects. The top-level group ID key retrieves high-level group details like name, image, etc. Nested objects store participant lists, settings, and other group metadata.
Status Updates
WhatsApp also allows users to share status updates that disappear after 24 hours. Like other content, status updates are stored with a unique ID key and relevant metadata in the value portion.
However, status updates also have an expiration time stored in the database. A background process periodically queries for expired status updates and deletes them to maintain the 24-hour lifetime.
Media Storage
WhatsApp handles incredible amounts of media sharing every day. Photos, videos, audio messages, and other files must be efficiently stored for retrieval when attached to messages.
Media files sent via WhatsApp are assigned a media ID key and stored in a media repository separate from the main message database. Small thumbnail versions may also be generated for efficient transmission.
When a media file is attached to a message, only the media ID is stored with the message contents. The media repository is queried via this ID whenever the recipient retrieves the message.
Keeping media attachments separate optimizes the main message database for smaller, more efficient data. It also allows WhatsApp to efficiently manage the lifecycle of media files.
Media Download and Caching
To optimize performance, WhatsApp maintains caches of media files on user devices and CDN edges near users. This avoids repeatedly downloading the same shared media.
A media file’s ID is hashed to check the local device cache when a message is opened. If a cache miss occurs, the media repository is queried and the file is downloaded and cached for future access.
User Privacy
User privacy is a major priority for WhatsApp. All chats are end-to-end encrypted by default, and private keys are only stored on user devices.
The encryption keys for chats are exchanged out-of-band when a chat is initiated between two users. WhatsApp’s servers only ever store encrypted message contents.
In group chats, encryption keys are managed by one designated user’s device. Keys are rotated periodically or when members join/leave to maintain forward secrecy.
Media attachments are encrypted in transit but stored unencrypted in WhatsApp’s media repository for efficiency. Users can enable enhanced media encryption for extra security.
Backups
WhatsApp allows users to backup their local message history to external services like iCloud or Google Drive. This provides a recovery mechanism in case a user loses their device or switches to a new one.
Backups contain a copy of the user’s local message database encrypted with a user-provided password. When restoring from a backup, the user provides the password to decrypt and import their message history.
Backups do not contain users’ private encryption keys. These keys have to be re-created when restoring so that previously encrypted messages can be decrypted by the user’s new device.
Scaling Challenges
WhatsApp’s continued growth presents enormous scaling challenges for its database architecture. From 2020 to 2021 alone, WhatsApp doubled its user base to 2 billion monthly active users.
To keep up with increasing demand, WhatsApp leverages Erlang’s distributed clustering capabilities. Erlang allows WhatsApp to horizontally scale its database across hundreds of nodes and distribute load.
WhatsApp also employs aggressive caching, data sharding, and geo-distribution techniques. Chat data is partitioned across geographic regions so each regional cluster only has to handle a subset of users.
Optimization of storage formats, network protocols, and client-side caching also helps minimize resource requirements for each additional user. Support for multiple device connectivity per user further increases the scaling complexity.
Reliability and Availability
As one of the most used apps worldwide, WhatsApp has to architect its systems for maximum reliability and availability. Billions of users expect 24/7 real-time messaging and 100% uptime.
To meet these expectations, WhatsApp employs redundant storage across clustered database nodes. Critical databases are replicated across multiple data center regions as well. Load balancers distribute queries across replicated nodes.
WhatsApp also carefully manages system upgrades, performing rolling updates to avoid large-scale downtime. Extensive monitoring provides alerts for any instability arising from updates or sudden spikes in traffic.
Despite these efforts, WhatsApp still suffers occasional outages, especially during major version upgrades. But its overall architecture continues maturing toward supporting reliable global-scale operation.
Conclusion
WhatsApp’s database architecture has scaled remarkably well, now supporting real-time messaging for over 2 billion users. The schemaless NoSQL structure based on Erlang provides the low-latency performance and horizontal scalability required.
As WhatsApp’s developers continue evolving its architecture, the focus remains on efficiency, reliability, and performance at massive global scale. Future growth will present even greater challenges, but WhatsApp’s infrastructure provides a solid foundation for the road ahead.