Search for a command to run...
I am a Journalist-turned-Software Engineer. I love coding and the associated grind of learning every day. A firm believer in social learning, I owe my dev career to all the tech content creators I have learned from. This is my contribution back to the community.
No comments yet. Be the first to comment.
Context: The objective of this exercise is to create a simplified version of Redis in C# as a learning project. We will call it "KVLite"! While Redis is a complex and feature-rich key-value store, we will focus on implementing a subset of its featur...
📖 Introduction In modern AI applications, communication between clients and tools isn’t always as simple as calling an API. The Model Context Protocol (MCP) provides a standardized way for clients to exchange information, invoke tools, and maintain ...
If you’re curious about how to build an intelligent agent using Model Context Protocol (MCP), you’re in the right place. In this post, I’ll walk you through how to: Create an MCP Server using FastMCP Expose a tool that calculates BMI Build a Clien...
In our previous tutorial, we built a restaurant customer support chatbot using OpenAI's Agents SDK. In this follow-up, we’ll explore guardrails—a critical feature that enhances AI chatbot safety and reliability. What Are Guardrails in AI Agents? Guar...
Video https://www.youtube.com/watch?v=UPBMkFSJdBI The Problem at Hand Uber's data platform processes approximately 1.2 million interactive queries monthly, with 36% of these coming from the operations organization. This group—comprising engineers...
Video https://www.youtube.com/watch?v=FgRGwnpd2HY Google's recent white paper on "Agents" has created quite a buzz. The paper explores the concept of AI agents and delves into their architecture and potential. Let's break down what this white paper...
I am Zahiruddin Tavargere (Zahere). A social-learner, here to learn, share and grow with the tech community.
74 posts
I am Zahiruddin Tavargere (Zahere). A firm believer in social learning, I owe my dev career to all the tech content creators I have learned from - this is my contribution back to the community.
If you are not familiar with Discord, here’s a quick introduction.
Discord is a voice, video, and text chat app that's used by tens of millions of people to talk and hang out with their communities and friends.
The communities in Discord are called “Servers”.
A server is a collection of persistent chat rooms and voice channels.
Discord has over 350 million registered users and over 150 million monthly active users.
Discord has been storing trillions of messages in Scylla.
Their growth over the years has led them to migrate from Mongo to Cassandra to Scylla in the span of 5 years.
Let’s understand the problems and decision-making that led them to ScyllaDB.
Discord initially used MongoDB to store messages, but they faced several issues, including poor performance and scalability.
MongoDB's architecture was not designed for the high write throughput that Discord required.
As a result, in 2017, they migrated to Cassandra, which provided better write performance and scalability.
By 2022, the Cassandra ‘messages’ database grew almost 15x to 177 nodes with trillions of messages.
With this growth came some problems…
Unpredictable latency
Very expensive maintenance operations
On-call system under constant stress
The Discord team zeroed down on one major cause. Partition.
Discord partitioned messaged by the channel they’re sent in. In Cassandra, all messages for a given channel were stored together and replicated across three nodes.
This approach to partitioning had a performance pitfall.
A discord server with thousands of people sends an order of magnitude more messages than a server with a few hundred users. This leads to a scenario that the discord teams refers to as “Hot Partitions”.
Hot partitions are a common issue in distributed databases.
They occur when a single partition receives a disproportionate amount of traffic, causing performance issues.
Hot partitions can be caused by a variety of factors, including uneven data distribution, poor partition key design, and high write throughput.
The discord team observed that the hot partition affected latency across the entire database cluster.
As one channel and bucket pair received large traffic, latency in the relevant node would increase leading to a broader end-user impact.
The process of compaction in Cassandra would further add to the latency making the reads very expensive.
Also, the discord team would constantly battle with JVM’s garbage collector as the garbage collector pauses would cause significant latency spikes.
Discord wrote data services that sit between their API and database clusters, using Rust as the language of choice. Rust's awesome concurrency and libraries were a great match for the task.
The big feature of these data services is request coalescing, which significantly reduces traffic spikes against the database.
Request coalescing combines multiple simultaneous requests to the same resource into a single request going to the origin. If multiple requests come in at the same time, they will be automatically merged.
Discord also implemented consistent hash-based routing to their data services to enable more effective coalescing.
These improvements helped reduce the load on the database.
Here’s how the Discord team migrated trillions of messages from Cassandra to Scylla.
Provision a new ScyllaDB cluster using a super-disk storage topology.
Dual-write new data to both Cassandra and ScyllaDB while concurrently setting up ScyllaDB's Spark migrator, a tool for data migration.
They extended their data service library to perform large-scale data migrations, using Rust programming language, which reduced the estimated time from 3 months to 9 days.
The migration encountered a challenge when it got stuck at 99.9999% complete due to large ranges of tombstones in the last few token ranges of data. The team resolved this by compacting the token range, and the migration was successfully completed.
Automated data validation was performed by comparing a small percentage of reads from both databases, ensuring the data was accurately migrated. The ScyllaDB cluster performed well with full production traffic, while Cassandra experienced frequent latency issues.
Smooth operations: No more weekend firefights or node juggling.
Optimized resources: 72 ScyllaDB nodes with 9 TB each, twice the storage capacity.
Improved latency: Fetching messages faster with 15ms p99 latency.
Discord's message storage is a complex system that requires careful consideration of performance and scalability. By migrating to ScyllaDB, Discord was able to solve their hot partition issues and improve their overall performance.
I write about System Design, UX, and Digital Experiences. If you liked my content, do kindly like and share it with your network. And please don't forget to subscribe for more technical content like this.
Below are some of my popular posts...