It is not a checklist of technologies. It is not whether you mentioned Kafka or Redis. Every interviewer at a serious tech company is evaluating you across a set of dimensions, and the expected depth on each dimension changes based on the level you are interviewing for.

The problem is that nobody publishes these rubrics. Companies keep them internal. So candidates end up guessing what “Senior-level depth” means versus “Mid-level depth,” and they calibrate against blog posts and YouTube videos that are often written by people who have never actually conducted these interviews.

What follows is a concrete rubric. No hand-waving. No “it depends.” For each dimension of a system design interview, here is exactly what is expected at Junior, Mid-Level, Senior, and Staff+ levels.

How to Use This Rubric

Before we dive in, a few things to understand about how interview leveling works.

Levels reflect interview expectations, not years of experience. A sharp engineer with 3 years of experience can interview at Senior level if they have worked on the right problems. A 10-year engineer can interview at Mid-Level for a domain they have never worked in. The rubric describes what you need to demonstrate, not who you need to be.

Key insight: Interviewers are not looking for you to hit every single point at your target level. They are looking for a pattern. If you consistently demonstrate Senior-level thinking across most dimensions, you will get a Senior rating even if you miss a few specifics.

The rubric covers nine dimensions. Not every interview will test all of them. A 45-minute round might only touch five or six. But understanding all nine helps you recognize what the interviewer is probing for and adjust your depth accordingly.

Subscribe now

1. Communication and Structure

This is the dimension that most candidates underestimate. It is also the one that interviewers evaluate from the very first minute.

At the Junior level, the bar is straightforward: state your approach before you start drawing, explain components as you add them, respond to questions clearly, and ask for feedback at major decision points. You are showing that you can think out loud and collaborate.

At the Mid-Level, you should propose a structured approach upfront. Something like: “I will start with requirements, then sketch the high-level design, then deep dive into the areas that matter most.” You time-box each section, summarize before moving on, and keep the whiteboard organized. This shows you have done this before.

At the Senior level, you drive the interview. You are not waiting for the interviewer to direct you. You adjust depth based on their signals. When you hit a decision point, you offer multiple options with trade-offs instead of just picking one. You proactively identify which areas deserve a deep dive.

At Staff+, you frame the problem before solving it. You teach while designing, explaining the “why” unprompted. You navigate ambiguity without needing guidance. And critically, you connect technical decisions to business outcomes. “We need eventual consistency here because the business can tolerate a 5-second delay on friend counts, and strong consistency would cost us 3x in latency.”

Why this matters so much: An engineer who builds a perfect system but cannot explain their reasoning is risky to hire. Communication is not a soft skill in system design. It is how interviewers determine whether you actually understand what you built or just memorized patterns.

2. Requirements and Estimation

This is where experienced engineers separate themselves from those who have only read about system design.

Junior: Ask 3-5 clarifying questions before designing. Identify core functional requirements. Confirm user types and basic use cases. Acknowledge that scale matters, even if you cannot calculate it precisely.

Mid-Level: Cover both functional and non-functional requirements systematically. Estimate DAU, QPS, and storage from the constraints given. Identify whether the system is read-heavy or write-heavy and explain the implications. Do basic math: QPS = DAU x actions per user / 86,400.

Senior: Do back-of-envelope calculations for storage, bandwidth, cache size, and server count. Identify latency SLAs (p50 and p99) and consistency requirements. Prioritize requirements as P0 versus P1 to scope the problem. Most importantly, use your estimates to justify design choices. “At 10K QPS, we need sharding because a single PostgreSQL instance tops out around 5K writes per second.”

Staff+: Challenge assumptions in the requirements. “Do we really need real-time updates, or would near-real-time with a 2-second delay be acceptable?” Estimate cost implications of design choices. Identify phased delivery milestones. Anticipate how scale will evolve: current state, 6 months out, 2 years out.

3. High-Level Design

The high-level design is where most candidates spend the majority of their time. Ironically, it is also where the level differences are most visible.

Junior: Show a clear client, API server, and database. Draw request flows with arrows. Identify the need for authentication. Demonstrate basic separation of concerns. This is the minimum viable architecture.

Mid-Level: Add a load balancer, stateless app servers, and database replicas. Include a cache layer with a clear invalidation trigger. Use a CDN for static content. Separate read and write paths when the access patterns justify it.

Senior: Define service boundaries with clear responsibilities. Introduce async processing via message queues for heavy operations. Match storage types to access patterns. Maybe you need a relational database for transactional data, a document store for user profiles, and a search index for full-text queries. Each choice should have a reason.

Staff+: Show multi-region topology with data flow. Identify single points of failure and their mitigations. Include observability integration points. Make deployment and rollback strategy visible in the design. At this level, the architecture tells a story about how the system operates, not just how it processes requests.

4. API Design

With the high-level architecture sketched, the next question is how services talk to each other and to the outside world. API design is often treated as an afterthought, but it reveals a lot about how a candidate thinks about system boundaries and contracts.

Junior: Define REST endpoints for core operations. Use correct HTTP methods. Sketch request and response structures. Mention authentication.

Mid-Level: Follow resource naming conventions. Include pagination with cursor or offset. Define a consistent error response format. Show awareness of rate limiting.

Senior: Design for idempotency on mutations using idempotency keys. Define a versioning strategy for API evolution. Choose appropriate async patterns: webhooks, polling, or streaming. Define API gateway responsibilities.

Staff+: Separate internal APIs from external APIs. Define backward compatibility guarantees. Analyze batch versus single-item trade-offs. Consider client SDK implications. At this level, you are designing APIs that other teams will build against for years.

5. Data Model

The data model is where interviewers probe your understanding of how data actually behaves at scale.

Junior: Identify main entities with attributes. Define primary keys. Show basic relationships (one-to-many, many-to-many). Use reasonable field types.

Mid-Level: Make a SQL versus NoSQL choice with reasoning. Define indexes for frequent query patterns. Explain denormalization decisions. Handle many-to-many relationships appropriately.

Senior: Select partition keys with reasoning. Prevent hot partitions. Consider read versus write optimized schemas (CQRS when appropriate). Define data lifecycle: TTL, archival, and deletion policies.

Staff+: Define cross-service data boundaries and ownership. Specify consistency guarantees at the entity level. Plan for schema evolution and backward compatibility. Address compliance considerations like PII handling, retention policies, and audit trails.

A pattern interviewers notice: Junior and Mid-Level candidates design data models that work. Senior candidates design data models that scale. Staff+ candidates design data models that evolve. The difference is in what you optimize for: correctness, performance, or longevity.

6. Scalability

Once the data model is defined, the interview typically shifts to how the system handles growth. Scalability questions reveal whether you have actually operated systems under load or just studied the theory.

Junior: Understand horizontal versus vertical scaling. Know that caching reduces database load. Recognize that stateless services can be replicated. Explain what a load balancer does.

Mid-Level: Use read replicas for read-heavy workloads. Implement cache-aside with TTL and invalidation strategy. Move work off the critical path with async processing. Use connection pooling for database efficiency.

Senior: Implement database sharding with consistent hashing. Use partitioned queues for parallel processing. Know write scaling strategies: sharding and batching. Implement back-pressure and load shedding to protect the system under stress.

Staff+: Design cell-based architecture to limit blast radius. Implement multi-region active-active with conflict resolution. Define a capacity planning methodology. Design graceful degradation tiers so the system loses features, not availability, under overload.

7. Reliability and Fault Tolerance

This is where production experience shows most clearly. Engineers who have been woken up at 3 AM think about failure differently than those who have not.

Junior: Understand that redundancy prevents single points of failure. Know that backups exist for data recovery. Retry on transient failures. Use timeouts to prevent hanging.

Mid-Level: Implement database replication for high availability. Use health checks with auto-restart. Apply exponential backoff with jitter on retries. Design graceful degradation when dependencies fail.

Senior: Discuss CAP trade-offs for specific components in your design. Use circuit breakers to prevent cascade failures. Make operations idempotent for safe retries. Define RTO and RPO requirements and explain how your design meets them.

Staff+: Isolate failure domains. Show awareness of consensus protocols for coordination. Define data durability guarantees including replication factor and sync versus async replication. Define SLIs that map directly to user experience, not just server metrics.

8. Trade-offs Discussion

Reliability and scalability are not free. Every design decision is a trade-off. The question is whether you recognize it and can articulate it.

Junior: Acknowledge that trade-offs exist. Identify at least one bottleneck. Explain why you chose your approach over alternatives. Be open to suggestions from the interviewer.

Mid-Level: Articulate 2-3 explicit trade-offs you made during the design. Compare alternatives with pros and cons. Identify the primary bottleneck and how to mitigate it. Explain your consistency versus availability choice.

Senior: Quantify trade-offs when possible. “This adds 50ms of latency but gives us 10x throughput.” Anticipate how bottlenecks shift at higher scale. Explain what you are NOT building and why. Factor operational complexity into your decisions.

Staff+: Do build versus buy analysis for components. Weigh engineering cost against system complexity. Consider the reversibility of decisions. Identify technical debt you are intentionally taking on and define the conditions under which it should be paid off.

The Staff+ signal: When a candidate says “this decision is easy to reverse if our assumptions are wrong, so I would start here and re-evaluate in 3 months,” the interviewer is hearing exactly what they want to hear. It shows maturity, pragmatism, and real-world judgment.

9. Deep Dive Ability

The deep dive is the final test. It is where interviewers push past your preparation and find the edges of your knowledge.

Junior: Explain your design choices when asked. Walk through a request end-to-end. Answer “what happens when X?” questions. Admit gaps in knowledge honestly. Honesty about what you do not know is always better than bluffing.

Mid-Level: Explain cache invalidation in detail. Walk through failure scenarios. Handle “what if component X fails?” questions. Be able to zoom into any component you drew and explain how it works internally.

Senior: Proactively offer to deep dive on complex areas without being asked. Explain edge cases like race conditions and split brain scenarios. Handle “what happens at 10x scale?” with specific, concrete changes to your design. Discuss operational concerns: how do you monitor this? How do you debug a slow request?

Staff+: Explain an incremental migration path from v1 to v2. Handle “what would you do with unlimited resources?” without falling into the infinite resources trap. Discuss how to validate the design before committing to a full build. Identify the biggest risks and define mitigation strategies for each.

Putting It All Together

Here is the pattern across all nine dimensions:

• Junior demonstrates awareness. They know the concepts exist and can apply them when prompted.

• Mid-Level demonstrates competence. They can apply concepts systematically and make reasonable choices.

• Senior demonstrates depth. They anticipate problems, quantify decisions, and think about operations.

• Staff+ demonstrates judgment. They challenge assumptions, think about evolution, and connect technical decisions to business outcomes.

The jump from Junior to Mid-Level is about breadth: covering more ground. The jump from Mid-Level to Senior is about depth: going further into each area. The jump from Senior to Staff+ is about judgment: knowing what matters and what does not.

If you are preparing for an interview, use this rubric to calibrate. Record yourself doing a mock interview, then score yourself on each dimension. The gaps will be obvious. Focus your preparation on the dimensions where you are below your target level, not on the ones where you are already strong.

And remember: the goal is not to memorize this rubric. The goal is to develop the thinking patterns that make these behaviors natural. When you genuinely understand why partition key selection matters, you do not need to remember that it is a “Senior-level expectation.” It just comes out because it is part of how you think about systems.

Score yourself against the full interactive rubric on System Overflow before your next interview. All 9 dimensions, all 4 levels, in one page. Identify exactly where to focus your preparation.

Learn system design @ System Overflow

OpenAI runs ChatGPT on a single PostgreSQL primary with 50 read replicas. One writer handling all writes for 800 million users.

Understanding why this works reveals something important about database scaling: the bottleneck you assume you have often isn’t the bottleneck you actually have.

Why Single-Primary Can Work

Subscribe now

The key insight is that ChatGPT’s workload is overwhelmingly read-heavy. Users send messages, but the system reads far more than it writes: fetching conversation history, loading user preferences, checking permissions, retrieving model configurations.

For read-heavy workloads, you don’t need to distribute writes. You need to distribute reads. And PostgreSQL’s streaming replication makes this straightforward: one primary handles all writes, and up to 50 read replicas serve read traffic across multiple geographic regions.

The math: With an overwhelmingly read-heavy workload and 50 replicas, each replica handles only a fraction of total read traffic. The primary focuses on writes. This is a fundamentally different scaling model than trying to distribute writes across shards.

This architecture delivers low double-digit millisecond p99 latency and five-nines availability. In the past 12 months, OpenAI has had exactly one SEV-0 PostgreSQL incident, and that was during the ImageGen launch when 100 million new users signed up in a single week.

The Real Bottlenecks

If single-primary works so well, why doesn’t everyone do it? Because making it work requires solving problems that most teams never encounter at smaller scale.

Connection limits. Azure PostgreSQL maxes out at 5,000 connections per instance. With hundreds of application servers, each maintaining connection pools, you hit this limit fast. OpenAI solved this with PgBouncer, a connection pooler that sits between applications and PostgreSQL. In transaction pooling mode, connections are returned to the pool after each transaction, allowing thousands of application connections to share hundreds of database connections. This dropped average connection time from 50ms to 5ms.

Cache miss storms. OpenAI uses a caching layer to serve most reads. But when cache hit rates drop unexpectedly, the burst of misses can overwhelm PostgreSQL. Their solution: cache locking. When multiple requests miss on the same cache key, only one request fetches from the database. The others wait for the cache to be repopulated. This prevents a single cache failure from cascading into a database outage.

Expensive queries. One 12-table join was responsible for multiple high-severity incidents. A spike in this single query could saturate CPU and slow down the entire service. The fix wasn’t just optimizing the query. It was recognizing that complex joins are an anti-pattern for OLTP workloads. If you need a 12-way join, break it into smaller queries and join in the application layer. Also: never trust your ORM. Always review the SQL it generates.

PostgreSQL’s MVCC Problem

While reads scale well, writes expose PostgreSQL’s fundamental limitation: its multiversion concurrency control (MVCC) implementation.

When you update a row in PostgreSQL, even a single field, the database doesn’t modify the existing row. It copies the entire row to create a new version. The old version becomes a “dead tuple” that remains in the table until vacuum cleans it up.

The cascade effect: Write amplification means you’re writing more data than you think. Dead tuples mean reads must scan past obsolete versions. Tables and indexes bloat, consuming more storage and slowing queries. Autovacuum struggles to keep up under heavy write loads, requiring careful tuning.

This is why OpenAI doesn’t try to scale writes on PostgreSQL. Instead, they’ve migrated write-heavy workloads to sharded systems like Azure CosmosDB. The workloads that remain on PostgreSQL are ones where read-heavy patterns make the single-primary architecture viable.

They’ve also banned adding new tables to PostgreSQL entirely. New features must use the sharded systems by default. This prevents the gradual accumulation of write-heavy workloads that would eventually overwhelm the primary.

Protecting the Primary

With only one writer, the primary is a single point of failure. OpenAI’s mitigation strategy has multiple layers.

Offload everything possible. Any read that doesn’t require transactional consistency with writes goes to a replica. This means that even if the primary fails, most user-facing requests continue working. Write failures are still serious, but the blast radius is smaller.

Hot standby. The primary runs in high-availability mode with a continuously synchronized standby ready to take over. Azure has optimized this failover to remain safe even under extremely high load.

Workload isolation. Requests are split into priority tiers and routed to separate instances. A new feature launch with inefficient queries won’t degrade the performance of critical requests. Different products are isolated from each other so that one product’s traffic spike doesn’t affect another.

Rate limiting everywhere. Limits are enforced at the application layer, connection pooler, proxy, and query level. The ORM layer can block specific query patterns entirely. When a surge of expensive queries hits, targeted load shedding allows rapid recovery without affecting other traffic.

Scaling Read Replicas

Adding read replicas seems simple: spin up more instances, point them at the primary, done. At OpenAI’s scale, it’s more complicated.

The primary streams Write Ahead Log (WAL) data to every replica. With 50 replicas, that’s 50 separate streams consuming network bandwidth and CPU on the primary. Each additional replica adds more pressure. You can’t scale replicas indefinitely without eventually overwhelming the writer you’re trying to protect.

OpenAI is working with Azure on cascading replication, where intermediate replicas relay WAL to downstream replicas instead of every replica connecting directly to the primary. This would allow scaling to over 100 replicas without proportionally increasing primary load. But it adds operational complexity around failover management, so it’s still in testing.

Schema Changes at Scale

In a normal PostgreSQL deployment, you might run ALTER TABLE without much thought. At OpenAI’s scale, schema changes are a production risk.

Some seemingly minor changes, like altering a column type, trigger a full table rewrite. The database locks the table, copies every row to a new version with the updated schema, then swaps the tables. For a table with billions of rows, this can take hours and block writes the entire time.

OpenAI’s rules are strict:

• Only lightweight schema changes are permitted (adding nullable columns, dropping columns that don’t trigger rewrites)

• Schema changes have a 5-second timeout. If it can’t complete in 5 seconds, it fails.

• Index creation must use CONCURRENTLY to avoid locking

• Backfilling table fields is rate-limited. Filling a new column can take over a week, but it doesn’t impact production.

When This Architecture Breaks Down

OpenAI’s approach works because their workload matches specific assumptions. Change those assumptions, and the architecture fails.

Write-heavy workloads don’t fit. If your application is 50% writes instead of 5% writes, a single primary becomes the bottleneck immediately. You need sharding or a database designed for distributed writes.

Strong consistency requirements complicate reads. Reading from replicas means accepting replication lag. If your application requires reading your own writes immediately, those reads must go to the primary, reducing the benefit of replicas.

Operational complexity is high. Managing 50 replicas across multiple regions, tuning PgBouncer, implementing cache locking, enforcing query patterns, rate limiting at every layer: this requires significant engineering investment. For smaller teams, a managed distributed database might be simpler even if it’s theoretically less efficient.

The Deeper Lesson

OpenAI’s PostgreSQL architecture isn’t a universal template. It’s an example of matching architecture to workload characteristics.

They could have sharded from the start. Many engineers would have assumed sharding was necessary at their scale. Instead, they analyzed their actual workload, recognized it was read-heavy, and built an architecture optimized for that pattern. The result is simpler than a sharded system (one writer, no distributed transactions, no cross-shard queries) while still handling 800 million users.

The lesson isn’t that sharding is bad or that single-primary is always better. It’s that scaling decisions should be driven by workload analysis, not assumptions about what “web scale” requires. The architecture that works depends on your read/write ratio, consistency requirements, query patterns, and team capacity.

Sometimes the boring solution, PostgreSQL with read replicas, scales further than you’d expect. You just have to solve the right problems.

Source: Scaling PostgreSQL to 800 Million Users - OpenAI Engineering Blog

Learn more about database scaling and system design @ System Overflow

This guide comes from hundreds of interviews at FAANG+ companies, on both sides of the table. No theory. No “draw these boxes.” Just what actually works.

What System Design Interviews Actually Test

Here is the truth. System design interviews are one of the best ways to interview a senior+ engineer. But they are also one of the most misused interview formats in our industry.

In some companies, system design has been reduced to a box-arranging exercise. Draw some rectangles, connect them with arrows, throw in a load balancer and a cache, and call it a day. This is not what a real system design interview looks like at serious tech companies.

At FAANG+ companies and for senior roles, system design tests both the breadth and depth of your knowledge. Done right, you cannot game this interview by memorizing solutions like you might for coding problems on LeetCode. Even if you have seen the exact question before, a skilled interviewer will find the gaps in your understanding.

The real test: System design interviews reveal whether you have actually built and operated systems at scale, or whether you have just read about them. You cannot fake experience when someone starts asking about failure modes and operational concerns.

This is what makes system design the best filter for identifying engineers with real working experience. The patterns of thought, the instincts about what can go wrong, the awareness of operational costs - these things only come from having been there and done that.

Master System Design Interviews

What to Expect in the Room

System design rounds typically run either 45 minutes or 60 minutes. Here is something critical that many candidates miss: there is no way anyone can finish a perfect system design in that time. Interviewers know this. They do not expect a flawless solution.

What they expect is a demonstration of how you think through problems. They want to see your process, your trade-off analysis, and your ability to make reasonable decisions under constraints.

Every interviewer has specific areas they care about. Maybe they are deeply interested in database choices. Maybe they want to explore caching strategies. Maybe they are focused on consistency guarantees. Your job is to pick up on these signals and engage with them.

Here is a mistake that shows up constantly: candidates come in with a prepared script. They have memorized a solution for “Design Twitter” or “Design Uber” and they recite it regardless of what the interviewer asks. By the end, the candidate feels great because they covered everything they prepared. Then they get rejected because they never gave the interviewer a chance to deep dive into the areas they actually cared about.

There is another problem with ignoring signals. Once you start digging into an area that the interviewer does not care about, it is very hard to come out of it cleanly. You lose time, you lose momentum, and you end up rushing through the parts that actually matter.

System design is a discussion, not a monologue. You are designing together with the interviewer, and you are leading that discussion. But you have to engage with them and build the solution collaboratively. If you just keep talking and dump everything you know, you will fail even if your technical knowledge is solid.

Time Management: The Silent Killer

Time management separates good candidates from great ones. Without a structure, you will either spend 30 minutes on requirements gathering and run out of time for the actual design, or you will rush through everything and miss critical depth.

Keep in mind that a 45-minute interview is not really 45 minutes of design time. You lose 2-3 minutes at the start for introductions and 2-3 minutes at the end for your questions about the company and team. So you are really working with about 40 minutes.

Here is a rough breakdown that works for most 45-minute interviews:

• Introductions: 2-3 minutes

• Understanding and requirements: 5 minutes

• High-level design (big picture): 10 minutes

• Deep dives: 20-22 minutes

• Your questions about the team/company: 2-3 minutes

For 60-minute interviews, you get more breathing room. Spend 8-10 minutes on requirements, 12-15 minutes on the big picture, and 30-35 minutes on deep dives. The extra 15 minutes mostly goes into deeper exploration.

The key insight here is that you need to know where you are at all times. If you are 20 minutes in and still discussing requirements, something has gone wrong. If you are 35 minutes in and have not started any deep dives, you are in trouble.

The Structure That Works

Having a structure is important, but do not treat it as the only way. You can adapt and improvise as long as you are confident about your approach. That said, here is the structure that works for most people.

Phase 1: Understanding the Question

Take 2-3 minutes, sometimes even 5 minutes, to just read and understand the question. Do not start talking immediately. Do not start drawing boxes. Just read.

This sounds obvious, but countless candidates miss critical parts of the question because they are too eager to start showing off their knowledge. The question often contains hints about what matters. A question that mentions “millions of users” is telling you that scale matters. A question that mentions “real-time” is telling you that latency matters. Read carefully.

Phase 2: Gathering Requirements

This is where many candidates go wrong. They either ask too many questions before showing any independent thinking, or they make too many assumptions without clarifying anything.

Here is the right approach: use a two-step process.

Step one: Write down your understanding of the functional requirements. List what you think the system needs to do based on the question. This shows the interviewer that you can analyze a problem and form your own understanding.

Step two: Ask clarifying questions about missing information. Now that you have demonstrated your thinking, ask about gaps and ambiguities.

Why this order matters: If you start asking questions before writing anything down, the interviewer has no way to judge your understanding of the problem. They do not want to spoon-feed you requirements. They want to see what you can figure out on your own first.

Do the same for non-functional requirements. State your understanding of what matters (latency, throughput, consistency, availability), then ask for clarification.

By the end of this phase, you and the interviewer should agree on both functional and non-functional requirements. This alignment is critical because everything else builds on it.

Reading the Signals

Here is something subtle but important. During requirements gathering, you will start to sense which parts of the problem the interviewer cares about.

Say you are designing a system like Pastebin. You ask about authentication requirements. If the interviewer says “any authentication is fine” or “not a primary concern for this problem,” that is a clear signal. They do not want you to spend time on authentication. They have something else in mind.

These signals are everywhere if you pay attention. When an interviewer leans in or asks follow-up questions, they are interested. When they give quick, dismissive answers, they want to move on. Learn to read these cues.

Phase 3: The Big Picture

The big picture is the skeleton of your design without getting into the details of anything. You are showing how requests flow, how data moves through the system, what components exist, but you are not making concrete technology choices yet.

Think of it as drawing a map before deciding which roads to take. You want the interviewer to see the overall shape of your solution before you start optimizing specific parts.

This phase should take 10-15 minutes. The key is to explicitly tell the interviewer what you are doing: “I am going to draw a high-level picture first to show the overall flow, and then we can dive deeper into specific components.”

This communication matters. The interviewer needs to understand your process. If you start drawing boxes without explaining your approach, they might think you are already in the details when you are just sketching the outline.

Phase 4: Deep Dives

This is where the real interview happens. By now, you either know which sections to explore deeply, or the interviewer will tell you explicitly.

But here is what separates good candidates from great ones: do not just wait for the interviewer to direct you. The most important thing that should come out of your system design interview is your proactive thinking.

The best area to show this is failure modes. Instead of waiting for the interviewer to ask “what happens if this service goes down,” you should identify failure scenarios yourself and present them.

This is what real senior+ engineers do. They do not wait for problems to be pointed out. They anticipate them.

The Infinite Resources Trap

This is the most common pitfall, especially from candidates who have read a lot but have not built much.

They will throw in every technology they have heard of. Kafka for messaging. Redis for caching. Elasticsearch for search. Cassandra for writes, PostgreSQL for reads. A separate analytics pipeline. Machine learning for recommendations. CDN for static content. Multiple regions for availability.

And they never stop to think: who is going to operate all of this? What happens when Kafka has a partition issue at 3 AM? How do you debug a request that touches seven different services? What is the cost of running all this infrastructure?

The reality check: An engineer who has spent sleepless nights on production incidents knows the importance of system reliability and manageability. An engineer who has only read about these systems will happily add complexity without understanding the operational cost.

Real experience shows in how you think about operations. Do you consider what happens when things fail? Do you think about the on-call engineer who will debug this at 2 AM? Do you understand that every additional component is another thing that can break?

This is why system design interviews are so effective at identifying real experience. You cannot fake this perspective. An engineer trying to portray themselves as experienced without having actually been there will fail. Either you have been burned by complexity and learned to respect simplicity, or you have not. There is no middle ground.

What Changes at Each Level

System design interviews are calibrated differently depending on the level you are interviewing for. Understanding these differences helps you focus on what matters.

Software Engineer (SWE)

At this level, interviewers want to see that you understand the basics. Can you break down a problem? Do you know what a load balancer does? Can you explain why you need a cache?

The expectations are:

• Clear problem decomposition

• Understanding of basic components (databases, caches, queues)

• Ability to make simple trade-off decisions

• Awareness that scale and reliability matter

You are not expected to design a globally distributed system. You are expected to show foundational knowledge and the ability to reason through problems.

Senior Software Engineer (SSE)

Now the bar goes up. You should be able to design a complete system end-to-end. Trade-off discussions become more important. You need to explain why you chose PostgreSQL over Cassandra, not just pick one randomly.

The expectations are:

• End-to-end system design capability

• Clear articulation of trade-offs

• Understanding of scaling strategies

• Awareness of operational concerns

• Ability to estimate capacity and identify bottlenecks

At this level, interviewers start probing your depth. They want to see that you have actually worked with these systems, not just read about them.

Staff Engineer

Staff level is where quality of judgment becomes the primary evaluation criteria. This is what you are paid for at this level, not how much code you write or how fast you ship.

There will be scenarios where you have to choose between X and Y, and the right answer is not obvious. Maybe both options have significant trade-offs. Maybe the data is incomplete.

It is perfectly fine to say “I do not know which is better at this point, but I would collect more data on A, B, and C before deciding. Based on what we know now, I am leaning toward X because...”

This is what interviewers want to see. They want to know how you handle ambiguity and make decisions with incomplete information.

The expectations are:

• Excellent judgment on complex trade-offs

• Ability to handle ambiguity gracefully

• Understanding of organizational and team impacts

• Proactive identification of risks and failure modes

• Clear communication of reasoning

Senior Staff and Principal

At this level, the scope expands beyond the system itself. You are expected to think about multi-year technical strategy, cross-team dependencies, and organizational implications.

Interviewers might ask questions like: “How would you migrate from the current system to this new design?” or “What team structure would you need to build and maintain this?” or “How does this fit with the company’s broader technical direction?”

The expectations are:

• Strategic thinking about technical direction

• Understanding of multi-team coordination

• Ability to design for long-term evolution

• Migration and adoption strategy

• Influence and alignment skills

The system design itself is almost secondary. What matters is how you think about building technology organizations and making decisions that affect hundreds of engineers.

What a Good Interviewer Looks Like

Knowing what a good interviewer does helps you recognize a fair interview and adjust when you are not getting one.

They guide without giving answers. If you are stuck, a good interviewer will drop hints or ask clarifying questions to nudge you forward. They will not let you flounder in silence, but they will not hand you the solution either.

They adapt to your level. A good interviewer is not trying to prove how hard the question is. They are trying to find where your knowledge ends. If you are clearly handling something well, they will push deeper. If you are struggling, they might simplify or redirect.

They tell you what they want. Vague interviewers who give no feedback and just watch you struggle are not running a fair evaluation. Good interviewers will say things like “let us focus on the database layer” or “walk me through the failure scenarios.” They give you direction.

They take notes instead of judging in real-time. If an interviewer seems distracted or checked out, that is a red flag. Good interviewers are engaged, listening, and writing things down.

Subscribe now

Common Mistakes to Avoid

The same mistakes show up over and over again. Here is what to avoid:

1. Starting to draw before understanding the problem. Take time to read and think. The eager candidate who starts drawing immediately often misses critical requirements.

2. Asking questions without showing your own thinking. Always demonstrate your understanding first, then ask for clarification. Do not expect the interviewer to define the problem for you.

3. Ignoring the interviewer’s signals. If they seem uninterested in a topic, move on. If they keep asking follow-up questions, go deeper. Read the room.

4. Treating it as a presentation instead of a conversation. System design is collaborative. Engage with your interviewer. Ask for their input. Respond to their questions.

5. Assuming infinite resources. Every technology choice has costs. Every additional component adds complexity. Show that you understand operational reality.

6. Not managing time. Know where you should be at each point in the interview. If you are running behind, adjust.

7. Avoiding saying “I don’t know.” Pretending to know something you do not know is worse than admitting uncertainty. Good engineers know their limits.

8. Forgetting about failure modes. Production systems fail. Show that you think about what happens when things go wrong.

Preparation That Actually Works

Everyone asks how to prepare for system design interviews. Here is what actually works, based on patterns from successful candidates.

Build things. There is no substitute for actual experience. If you have never operated a system at scale, you will have blind spots that preparation cannot fill. Side projects, open source contributions, or taking on infrastructure work at your current job all help.

Read postmortems. Companies publish detailed analyses of outages. These are gold mines for understanding how real systems fail and how experienced engineers think about reliability.

Study real architectures. Many companies have published blog posts about their systems. Read about how Netflix handles streaming, how Uber manages rides, how Slack handles messaging. Understand the decisions they made and why.

Practice with mock interviews. Find someone to practice with. The experience of explaining your thinking out loud, under time pressure, with someone asking questions, is very different from thinking through a problem alone.

Focus on fundamentals. You do not need to know every database and every message queue. You need to deeply understand when to use different types of storage, how to handle consistency and availability trade-offs, how caching works, and how to scale systems.

The Bigger Picture

System design interviews exist because building software at scale is hard. The problems you encounter at 10,000 users are different from 10 million users. The decisions you make early can haunt you for years. The cost of getting it wrong is measured in downtime, lost revenue, and frustrated users.

What interviewers are really trying to understand is: can this person make good decisions about complex systems? Can they reason about trade-offs? Do they understand what they do not know? Will they build something that works, or something that looks good on a whiteboard but falls apart in production?

The engineers who do well in these interviews are the ones who have learned from experience. They have seen things break. They have debugged production issues at 3 AM. They have dealt with the consequences of poor architectural decisions. That experience shows in how they think and what they worry about.

You cannot fake experience. But you can build it.

How? Start paying attention to how the systems you work with actually behave. Ask questions about why things were built the way they were. Volunteer for on-call rotations. Read internal incident reports. Over time, you will develop the instincts that make system design interviews feel like natural conversations instead of tests.

That is the goal. Not to memorize answers, but to develop genuine understanding. When you have that, the interview stops being a test and starts being a conversation about systems you actually know how to build.

Master System Design @ System Overflow

Meta built DrP to solve this exact problem. Over five years in production, processing 1.5 million analyzer runs and 250K alerts every 30 days, DrP has reduced mean time to resolve (MTTR) by 20% on average. Teams that comprehensively adopted the platform achieved 50-80% reductions. But DrP isn’t just about automation. It’s a case study in how to capture tribal knowledge, build extensible investigation workflows, and create a system that compounds value as more teams adopt it.

You’ll learn how Meta approached the core challenges of automated incident investigation: building an expressive SDK for diverse investigation patterns, scaling execution to handle thousands of concurrent analyses, and integrating seamlessly into existing workflows.

The Challenge: Investigation Workflows That Don’t Scale

Meta’s infrastructure spans thousands of services with complex dependencies. When a metric degrades, the investigation typically follows a decision tree. Is it a configuration change? Did a dependency fail? Is it isolated to one region? Traditional approaches fail at this scale for three reasons.

First, playbooks become outdated immediately. Engineers document investigation steps in wikis or runbooks, but these grow stale as systems evolve. A service that used to call three dependencies now calls twelve. The configuration format changed. By the time the next incident hits, your carefully documented playbook leads to dead ends.

Second, ad-hoc scripts don’t compose. Individual teams write Python scripts or SQL queries to automate parts of their investigations. But these scripts are point solutions. They hardcode assumptions about data locations, assume specific metric names, and can’t be chained together. When you need to investigate a dependency’s dependency, you’re back to manual work.

At scale, the real problem isn’t any single incident. It’s the combinatorial explosion of investigation paths across hundreds of services.

Think of it like a choose-your-own-adventure book where every page is written by a different author using a different language. You can’t jump between chapters. Each investigation starts from scratch.

Third, tribal knowledge remains locked in engineers’ heads. Your senior engineer knows that when metric X drops, you should check configuration Y in region Z. But that knowledge disappears when they’re on vacation or leave the team. New on-call engineers spend hours rediscovering patterns that experts recognize instantly.

Meta’s Solution Architecture

Subscribe now

DrP addresses these challenges through a platform approach rather than a tool approach. Instead of replacing human investigation, it provides infrastructure to capture, scale, and automate investigation workflows.

The SDK: Codifying Investigation Logic

At the core is DrP’s SDK, which lets engineers author analyzers (automated investigation playbooks) in Python or PHP. The SDK provides a Context class, a key-value dictionary for storing investigation parameters like alert ID, service name, threshold violations, and telemetry data locations. Input APIs capture and validate these parameters, while any inferred values can be dynamically added during execution.

Critically, DrP provides declarative, strongly typed APIs to query data sources including time series databases, analytical databases, data warehouses, and log databases. Instead of writing raw SQL strings that are difficult to maintain and debug, engineers use type-safe APIs tailored to common investigation patterns. This eliminates hardcoded queries and enables easy reuse across analyzers.

After analysis completes, outputs are captured in a structured Findings class that supports flexible rendering and evidence inclusion. Findings can output plain text or machine-readable formats including Thrift payloads with self-describing schemas. These structures facilitate metadata addition for standardized UI widgets and custom React components, enabling downstream processing and analytics.

Analysis Libraries: Scaling Investigation Intelligence

Services generate massive amounts of observability data. DrP includes scalable analysis algorithms based on statistical and ML techniques:

Dimensional analysis helps isolate issues across multiple dimensions (region, endpoint, cluster). For the most frequent repeated investigations, a pre-aggregation layer can reduce dataset size by up to 500X, significantly speeding up real-time, latency-sensitive investigations.

ML-based event isolation addresses a common cause of incidents: code changes and config deployments. The library ranks thousands of code and config change events using signals like text matching, time correlation with alerts, and on-call context. On average, this filters out the majority of uninteresting events, surfacing the most suspicious ones with confidence annotations explaining the ranking.

Time series correlation helps identify relationships between metrics across different services and data sources.

The analysis libraries provide both rule-based and ML-based techniques, but pure ML systems have limitations in data quality and customization. Meta learned that combining rule-based suggestions from community expertise with AI, augmented by dashboards for visualization, works better than purely automated approaches.

Analyzer Chaining: Composing Investigations

Services depend on other services. When investigating a frontend issue, you often need to check if backend dependencies are healthy. DrP solves this with analyzer chaining, allowing analyzers to call other analyzers in a sequence or DAG (Directed Acyclic Graph).

Key capabilities include: passing inputs and context to dependent analyzers with temporary overrides for additional parameters; flexible outputs via the Findings class that calling analyzers can parse for relevant information; and lazy import of analyzers for dynamic chaining without upfront latency. Cross-platform support allows chaining between PHP and Python analyzers.

This promotes analyzer reuse, with over 21% of analyzers using chaining. Using lines of code as a measure, analyzer chaining provides 3x improvement for typical service debugging use cases. Power users report 5-10x faster development after migrating from ad-hoc solutions.

The Scalable Backend

DrP maintains 99.9% backend availability while processing massive scale. The backend uses a MySQL-backed queue store with fields for request ID, timestamp, analyzer identifier, context, and status. Worker tiers run executors that parse requests and run analyzers in sandbox environments, with results returning asynchronously via callbacks.

A key challenge: with 2000+ analyzers constantly in churn, packaging all into one binary increases size and load time, and creates noisy neighbor issues where one analyzer failure affects others. Instead, DrP creates smaller analyzer groups based on expected affinity, each with its own binary.

When a request arrives, the system identifies the analyzer group, dynamically fetches the binary, and launches it as a subprocess. Since 85% of traffic comes from the top 10% of analyzers, those binaries are pre-loaded at startup while others are lazy-loaded. For the most frequently used analyzers, they’re embedded directly in the executor binary for negligible overhead. This provides a balance of quick startup and acceptable delays for dynamic fetching.

Workflow Integration: Bringing Analysis to Engineers

DrP integrates directly into Meta’s alerting and incident management systems. When an alert triggers, it auto-executes associated analyzers without engineer intervention. Investigation results appear immediately on the alert page, providing context before the on-call engineer even acknowledges the page. The process from alert to analysis output typically takes a few minutes, providing near real-time analysis.

This integration is crucial. If engineers need to log into a separate system, select an analyzer, and wait for results, adoption suffers. By embedding DrP into existing workflows, Meta ensures the path of least resistance is using automated investigation. Additionally, a standalone UI and CLI are available for ad-hoc investigations, used by over 450 unique users per week.

Post-Processing: Closing the Loop

After analysis completes, DrP’s post-processing tier executes automated actions based on results: creating incident tickets with pre-filled context, generating PRs to fix configuration drift, annotating alerts with likely root causes, or triggering remediation workflows.

The DrP Insights system periodically analyzes historical analyzer outputs to identify and rank top alert causes, helping teams prioritize reliability improvements. Instead of just reacting to individual incidents, teams can see aggregated patterns and invest in fixing root causes.

Quality Assurance: The Backtesting Framework

Testing analyzers is challenging. There’s no good way to record past incidents, and unit tests miss coverage due to dynamic investigation paths. Meta developed a novel backtesting mechanism: they retain inputs and outputs from past analyses (default 30 days), enabling integration tests on historical data for modified analyzers.

These tests filter out non-logic errors, highlighting actual code change issues. Automated in the PR review process, they block PRs until errors are fixed. Combined with canary testing that runs a sample of production traffic before deployment, this has greatly improved analyzer quality and prevented deployment of buggy analyzers.

Trade-offs and Lessons Learned

Assist versus full automation: Meta initially aimed for complete automation but pivoted to an assistive approach. Statistical and ML analysis have limitations and produce false positives. Engineers don’t always trust fully automated systems. And keeping analyzers updated with evolving systems is difficult. DrP now offers insights and recommendations that engineers validate, balancing automation with human expertise.

Adoption depth matters more than usage. Teams with fewer than 5 analyzers see 10-15% MTTR improvement. Teams with 10+ analyzers consistently achieve 50-80% reductions. One team reduced MTTR from 771 hours to 139 hours (82% improvement) after building 136 analyzers.

Data quality is everything: Seamless investigation requires quality data and metadata in telemetry systems. Meta faced issues correlating data from different sources and lacked structured metadata for service dependencies or data lineage, limiting downstream correlations.

Community-driven development: DrP’s success came from democratizing analyzer development. The team bootstrapped adoption by building analyzers for common investigations (services infrastructure, hardware), then developed the SDK for teams to build custom analyzers and chain them together. The engaged community effort was essential for scaling across 300+ teams.

Analyzer maintenance: Like any software, analyzers need long-term maintenance. Teams need to estimate the right time to invest. If investigations are simple and repetitive, dashboards may be more effective. As complexity and team size grow, analyzers add more value.

System Design Patterns Worth Noting

DrP demonstrates several architectural patterns that appear frequently in system design:

Workflow orchestration with DAGs: The analyzer chaining model mirrors how systems like Airflow or Temporal handle complex workflows. Each analyzer is a node that can invoke dependencies, pass context, and aggregate results. This pattern appears whenever you need to coordinate multi-step processes with dependencies.

Queue-based async processing: The MySQL-backed queue with worker pools is a classic pattern for handling variable load. Requests are decoupled from execution, allowing the system to handle bursts (like widespread incidents triggering hundreds of alerts) without blocking callers.

Lazy loading for scale: Pre-loading the top 10% of analyzers that handle 85% of traffic while lazy-loading the rest is a practical application of the Pareto principle. This pattern applies to any system with skewed access patterns, from CDN caching to database connection pools.

Structured output contracts: The Findings class with Thrift schemas enables loose coupling between analyzers and consumers (UIs, post-processors, other analyzers). This is the same principle behind API contracts and schema registries in event-driven architectures.

Based on Meta Engineering blog: DrP: Meta’s Root Cause Analysis Platform at Scale

Master system design concepts @ System Overflow

Dropbox faced this problem with Dash, their conversational AI that answers questions about files, wikis, and connected tools. Their solution was to treat evaluation as infrastructure, not an afterthought. This article breaks down how they designed it.

Subscribe now

Why Traditional Metrics Failed

Dash is not a single model. It is a pipeline:

user query → intent classification → document retrieval → ranking → prompt construction → LLM inference → safety filters

Each stage is non-deterministic. Changing retrieval parameters alters which documents reach the model. That interacts with the prompt template. Which affects how often the model cites sources correctly. You cannot reason about quality by looking at any component in isolation.

Early on, Dropbox engineers relied on classic NLP metrics: BLEU, ROUGE, BERTScore, embedding similarity. These measure surface overlap or semantic proximity, not production correctness.

The core problem: A high ROUGE score can hide a hallucinated filename. Strong embedding similarity can correspond to an answer that ignored the question. These metrics cannot answer: did the response actually come from the user’s documents? Were claims supported by retrieved context? Were the correct files cited?

As soon as Dropbox tried to use these metrics to gate real deployments, they broke down.

The Flight Control Analogy

Consider how you would certify a new autopilot system. You would never approve it only by checking that steering roughly matches a reference trajectory. You also need alarms for altitude, fuel, structural stress, and dozens of other dimensions.

An LLM evaluation system works the same way. It must be a network of alarmed checks, not a single similarity score. Quality is multi-dimensional and context-dependent, but your system must compress it into automated checks that decide whether a change can ship.

Dropbox had additional constraints:

• Evaluate on both public benchmarks and Dropbox-specific content

• Every model, retriever, or prompt change must behave like production code with tests wired into CI

• Scale evaluation to many experiments without drowning in manual labeling

The Three-Layer Architecture

Dropbox built an evaluation platform that wraps the LLM application. It has three layers: curated datasets, LLM-powered metrics, and a CI-integrated orchestration platform.

Upgrade to Pro - 10% EXTRA OFF

Layer 1: Dataset Curation

Dropbox combined two sources:

Public QA datasets. Natural Questions, MS MARCO, and MuSiQue stress-test retrieval, multi-document answers, and multi-hop reasoning. These provide reproducible baselines.

Internal datasets from Dash usage. From logs, they created representative query sets (anonymized, ranked real queries) and representative content sets (popular files, docs, connected sources). From that content, they generated synthetic questions and answers using LLMs, covering tables, images, tutorials, and factual lookups.

Together, these datasets approximate the long tail of real usage in a reproducible form you can re-run in every experiment.

Layer 2: LLM-as-Judge Metrics

Classic metrics still run as quick sanity checks. But the core of the system is LLM-as-judge.

A judge model receives four inputs: the user query, the candidate answer from Dash, the retrieved context, and optionally a hidden reference answer. It scores specific dimensions:

{
“factual_accuracy”: 4,
“citation_correctness”: 1,
“clarity”: 5,
“formatting”: 4,
“explanation”: “Answer was accurate but referenced a source not in context.”
}

Dropbox treats these judges like software modules: versioned, tuned against small human-labeled calibration sets, and periodically re-checked for agreement with human evaluators.

Key insight: Evaluating the evaluators becomes part of the loop. Judge models can drift or develop blind spots. Periodic human spot-checks keep them calibrated.

Layer 3: Evaluation Platform as CI

The platform works like CI for LLM behavior. A central store holds datasets and judge configurations. An orchestrator runs suites against any pipeline version or prompt template.

The test pyramid:

• Pull requests — Fast regression subset, blocks merge on failures

• Staging and nightly — Full curated suites

• Production — Sampled live traffic re-run through judges to monitor drift

The same evaluation logic runs everywhere, giving consistency and traceability across experiments and releases.

Enforcement Levels

Not all metrics are equal. Dropbox separates them into three tiers:

1. Boolean gates — Hard fails. Examples: “citations present”, “source file exists”. A single failure blocks deployment.

2. Scalar budgets — Thresholds that cannot regress. Examples: minimum source F1, p95 latency ceiling, cost per query. Changes that degrade these are blocked.

3. Rubric scores — Softer properties like tone, narrative quality, formatting. Tracked in dashboards but do not block deployment.

This separation prevents promising experiments from being blocked by minor rubric regressions while still protecting users from factuality and citation failures.

Trade-offs

Dropbox’s choices came with clear trade-offs.

LLM judges add cost and drift risk. They provide flexibility: you can encode complex rubrics and adapt to new tasks without training task-specific scorers. But they are another model that can degrade. Dropbox uses smaller, specialized judge models when possible and relies on human spot-checks for calibration.

CI integration slows iteration. Wiring evaluations into every pull request increases reliability but can bottleneck development if suites are too heavy. Dropbox addresses this with the test pyramid: tiny fast subsets for PRs, full suites for staging and nightly runs.

Hard gates can block good changes. A change might hurt one metric while improving others. Separating boolean gates, scalar budgets, and rubric scores gives flexibility. Safety-critical dimensions block; UX dimensions inform.

Requires representative data. This approach assumes you can collect internal usage data and have some human labeling capacity. In domains with sparse data or strict privacy rules, you may need heavier reliance on public datasets and synthetic generation.

When This Pattern Applies

Use evaluation-as-infrastructure when:

• Your LLM system has multiple interacting stages (retrieval, ranking, generation)

• Quality is multi-dimensional (accuracy, citations, latency, cost)

• You need to gate deployments on behavior, not just unit tests

• Multiple engineers iterate on prompts, models, and retrievers in parallel

• You have access to representative usage data or can generate synthetic queries

You can skip this complexity when:

• Your LLM usage is simple (single prompt, no retrieval)

• Quality can be measured with a single metric

• You have low deployment frequency and can manually review each change

The Takeaway

Dropbox’s work on Dash shows that for LLM applications, evaluation is core system design, not an afterthought. By curating realistic datasets, using LLMs as structured judges, and wiring evaluations into CI and production, they turned a fragile text box into a monitored, testable system.

The patterns connect directly to classic system design: test pyramids, observability, deployment gates. The difference is that LLM behavior is probabilistic, so your checks must be probabilistic too. Treat datasets as versioned assets. Design judge prompts as reusable modules. Wrap your pipeline in an evaluation platform that can say no to unsafe changes.

Reference: A practical blueprint for evaluating conversational AI at scale.
Learn more about ML System Design @ System Overflow

Understanding what happens inside the file is how you unlock that performance.

Why Storage Layout Matters

At the logical level, data is simple: tables with rows and columns. But how you physically arrange bytes on disk has massive performance implications.

Consider a typical analytics query: “What’s the average order value by country?” You need two columns out of a wide table with dozens of fields.

Row-wise storage (how traditional databases work) lays data out like this:

[row1_col1, row1_col2, ... row1_colN] [row2_col1, row2_col2, ... row2_colN] ...

To read your two columns, you scan through every column for every row. Most of that I/O is wasted on data you’ll immediately discard.

Columnar storage inverts the layout:

[row1_col1, row2_col1, row3_col1, ...] [row1_col2, row2_col2, ...] ...

Now reading two columns means reading two contiguous blocks. You only touch the data you actually need. Query engines call this projection pushdown, and columnar formats make it efficient.

Subscribe now

The tradeoff: Row-wise storage excels at transactional workloads (insert a record, update a field). Columnar storage excels at analytical workloads (aggregate millions of rows, but only a few columns). Different access patterns, different optimal layouts.

The Locality Problem with Pure Columnar

Imagine reconstructing a single row from a 100GB columnar file with 10 columns. Column A is in the first 10GB. Column B is 10GB away. Column J is 90GB away. You’ve lost all data locality.

Modern hardware is built around locality. CPU caches, memory prefetching, disk read-ahead - they assume if you read address X, you’ll want X+1 soon. Scattered access defeats these optimizations.

Parquet solves this with a hybrid model: divide rows into groups (typically 128MB), then store each group in columnar format. You get columnar benefits within each group, plus locality across groups.

Inside a Parquet File

A Parquet dataset is often a directory containing multiple .parquet files. Within each file:

Row Groups are horizontal partitions, typically 128MB each. They contain a subset of rows with all their columns.

Column Chunks are vertical partitions within a row group. Each chunk contains all values for one column in that group.

Data Pages are the actual storage units within column chunks - encoded values plus metadata like min/max statistics.

Footer stores all metadata: schema, row group locations, and statistics. Reading the footer first lets you plan which parts of the file to actually read.

How Parquet Compresses Data

Columnar layout enables compression techniques that don’t work well on row-wise data. When values from the same column sit together, they share the same type and often similar distributions.

Parquet applies three techniques in sequence:

1. Dictionary Encoding

Original: [”USA”, “France”, “Germany”, “Netherlands”, “Netherlands”, “Netherlands”]

Dictionary: {0: “USA”, 1: “France”, 2: “Germany”, 3: “Netherlands”}
Encoded: [0, 1, 2, 3, 3, 3]

Repeated string values become small integer references. Within each row group, a “country” column stores each unique value just once in its dictionary.

2. Run-Length Encoding

Before: [3, 3, 3]
After: (value=3, count=3)

Consecutive identical values collapse into (value, count) pairs. Especially powerful on sorted data where similar values cluster.

3. Bit Packing

4 unique values in dictionary = 2 bits needed per reference (not 32)

With a small dictionary, each reference needs only log2(dictionary_size) bits instead of a full integer.

Watch out: Dictionary encoding has a size limit per column chunk. If you exceed it (too many unique values), Parquet falls back to plain encoding and loses compression benefits. Fix this by increasing dictionary page size or decreasing row group size.

On top of encoding, Parquet supports page-level compression (Snappy, GZIP, LZ4). A benchmark from S3: reading 10GB uncompressed took ~14 seconds; the same data with Snappy compression took ~8 seconds. Smaller files mean less I/O, even accounting for decompression CPU cost.

Skipping Data with Statistics

Compression reduces file size. But the bigger wins come from not reading data at all.

Every row group stores min/max statistics in the footer. For a query like WHERE user_id > 1000, Parquet checks each row group before reading:

Row Group 0 (min: 0, max: 900)
→ SKIP - max value 900 is below 1000

Row Group 1 (min: 850, max: 2100)
→ READ - range overlaps with predicate

Row Group 2 (min: 1, max: 400)
→ SKIP - max value 400 is below 1000

Each skipped row group is 128MB you don’t read. On sorted data, row group skipping can eliminate most of your I/O.

Important: This only works well when data is sorted or clustered on the filter column. Randomly distributed data produces wide min/max ranges that overlap with most predicates, defeating the optimization.

For equality predicates (WHERE country = 'Germany'), Parquet offers dictionary filtering: check if the value exists in the column chunk’s dictionary before reading any data pages. If it’s absent, skip with certainty.

The Small Files Problem

Every Parquet file has overhead: connection setup, reader instantiation, footer parsing. With a few large files, this overhead is negligible. With thousands of small files, it dominates query time.

Benchmark (10GB dataset on S3):

16 files: ~12.5 seconds
1,024 files: ~19 seconds

Same data, 50% slower due to file overhead.

Small files accumulate naturally. Hourly ETL jobs create a new file each run. Streaming ingestion creates small batches. Partitioning on high-cardinality columns fragments data across directories. After months, you have tens of thousands of files.

The opposite extreme is also problematic. One team consolidated 250GB into a single file. A simple COUNT(*) went from 5 minutes to over an hour. The culprit: footer processing. Massive files have massive metadata (thousands of row groups), and Parquet’s footer parsing isn’t optimized for that scale.

Sweet spot: Target files in the 128MB to 1GB range. Compact small files periodically. Split oversized files.

Directory Partitioning

When you know your query patterns upfront, embed predicates in the directory structure:

/data/events/
  date=2024-01-15/
    part-00000.parquet
  date=2024-01-16/
    part-00000.parquet

A query filtering on date doesn’t open irrelevant directories. Predicate evaluation becomes file listing.

The tradeoff: partitioning on high-cardinality columns (user_id, timestamp) creates thousands of directories with tiny files. Use partitioning for low-cardinality columns you frequently filter on (date, region, status).

Practical Optimization Checklist

Reduce file size:

• Enable compression (Snappy is usually the right default)

• Monitor dictionary encoding fallback on high-cardinality columns

• Only SELECT columns you need - projection pushdown requires it

Enable data skipping:

• Sort data on commonly filtered columns for tighter min/max ranges

• Use typed predicates matching column types (avoid implicit casting)

• Partition by low-cardinality, frequently-filtered columns

Manage file count:

• Compact small files from incremental writes

• Avoid single massive files (footer parsing bottleneck)

• Consider Delta Lake or Iceberg for automatic compaction and better metadata handling

The Takeaway

Parquet’s performance comes from a set of deliberate tradeoffs: columnar layout for projection pushdown, encoding for compression, statistics for skipping. These optimizations are automatic, but they only work well when your data organization matches your access patterns.

Sort your data on filter columns. Keep file sizes reasonable. Select only the columns you need. The format will do the rest.

Subscribe now

Go deeper on data engineering fundamentals @ System Overflow

Why Data Engineering?

Modern applications generate massive data volumes. Whether you’re building real-time analytics, ML pipelines, or data warehouses, you need to design systems that can ingest, transform, and serve data reliably at scale.

Data Engineering interviews test your ability to make design decisions under constraints. System Overflow focuses on the trade-offs and patterns that matter in real interviews.

What’s Inside the Track

🎯 12 Core Areas • 90+ Topics • 400+ Learning Cards

Data Modeling & Schema Design
Dimensional modeling, normalization trade-offs, time-series patterns, slowly changing dimensions

Data Pipelines & Orchestration
DAG-based orchestration (Airflow, Prefect), idempotency, backfills, cross-pipeline dependencies

Storage Formats & Optimization
Parquet/Avro/ORC internals, compression algorithms, encoding strategies, partitioning patterns

Batch vs Stream Processing
Lambda/Kappa architectures, micro-batching, hybrid processing models

Distributed Data Processing
Spark execution model, Catalyst optimizer, distributed joins, shuffle optimization, memory tuning

Stream Processing Architectures
Kafka Streams, Flink state management, windowing, exactly-once semantics, watermarking

Data Lakes & Lakehouses
Delta Lake/Iceberg/Hudi internals, ACID transactions, metadata catalogs, table formats

Change Data Capture (CDC)
Log-based CDC (binlog, WAL), consistency guarantees, performance at scale

Real-time Analytics & OLAP
Druid/ClickHouse architecture, pre-aggregation patterns, approximate query processing

Data Quality & Validation
Schema validation, data contracts, anomaly detection, reconciliation techniques

ETL/ELT Patterns
Incremental processing, transformation layers (bronze/silver/gold), dbt workflows, deduplication

Data Governance & Lineage
Lineage tracking, access control, data masking, GDPR compliance, catalog systems

How It Works

Every topic includes:

✅ Expert-curated content with depth that matters in real interviews
✅ Trade-off analysis for making informed design decisions
✅ Practical scenarios from actual production systems
✅ Progressive difficulty with time estimates
✅ Implementation patterns that work at scale

This isn’t just theory. It’s the mental models you need to design data systems that handle billions of events per day.

Who This Is For

📌 Senior/Staff/Principal Engineers preparing for data infrastructure roles
📌 Backend Engineers moving into data platform teams
📌 Data Engineers leveling up their system design skills
📌 Anyone building pipelines, warehouses, or real-time analytics at scale

Get Started Today

👉 www.systemoverflow.com

Join engineers from FAANG+ companies using System Overflow to level up their design skills.

10% OFF - Buy Pro

Already crushing System Design or ML Design on the platform? The Data Engineering track is waiting for you.

New to System Overflow? Start with any track. All three are designed to work together as you build end-to-end expertise.

Let’s design better Systems. 🚀

To understand TPUs, we need to start with a question: what do neural networks actually do at the hardware level?

Subscribe now

The Problem TPUs Solve

Neural network training and inference are dominated by one operation: matrix multiplication. Forward pass? Multiply weight matrices by activation vectors. Backpropagation? More matrix multiplies to compute gradients. Attention mechanisms? Matrix multiplies between queries, keys, and values.

GPUs handle this reasonably well because they’re designed for parallel computation. But GPUs are also designed to run graphics, physics simulations, cryptocurrency mining, and general-purpose computing. All that flexibility comes with overhead: complex instruction decoding, cache hierarchies for unpredictable memory access, branch prediction for conditional code.

Google asked: what if we threw all that away and built hardware that only does matrix multiplication?

The Systolic Array: TPU’s Core Innovation

The answer is the systolic array, a computing architecture from the 1980s that fell out of favor for general computing but turns out to be perfect for matrix math.

Picture a two-dimensional grid of simple processing elements. Each cell does exactly one thing: multiply two numbers and add the result to a running total. No conditionals, no branching, no complex logic.

How data flows: Weights load into the array and stay stationary. Activations flow in from the left, moving one cell right each clock cycle. Partial sums flow downward. By the time data exits the bottom of the array, you have your matrix multiplication result.

The term “systolic” comes from the heart. Just as blood pulses through your circulatory system in waves, data pulses through the array in a regular, predictable rhythm.

This predictability is the key insight. Because data flow is fixed, the hardware never waits for memory, never mispredicts a branch, never stalls on a cache miss. Every cycle, every cell is doing useful work.

Even better, each value gets reused multiple times. A weight sitting in a cell multiplies against every activation that flows past it. An activation moving across a row multiplies against every weight in that row. This data reuse dramatically reduces memory bandwidth requirements.

What TPUs Give Up

This efficiency comes from extreme specialization. Understanding what TPUs can’t do is just as important as understanding what they excel at.

No general-purpose computing. TPUs can’t run arbitrary code. They only execute tensor operations. Anything else, from data preprocessing to custom logic, runs on a host CPU and communicates with the TPU over PCIe.

No CUDA, no direct programming. You don’t write TPU kernels. Instead, you write TensorFlow or JAX code, and the XLA (Accelerated Linear Algebra) compiler transforms it into TPU instructions. This means automatic optimization, but also less control when things don’t compile efficiently.

No flexibility on precision. TPUs are designed around bfloat16, a 16-bit format optimized for ML. This works well for neural networks, but if your workload needs higher precision, you’re fighting the hardware.

No purchasing or on-prem deployment. TPUs are cloud-only, rented from Google Cloud. You can’t buy them for your own datacenter.

How This Compares to GPUs

Now that we understand how TPUs work, the comparison to GPUs becomes clearer. These aren’t just different products; they’re different philosophies.

1. Parallelism model — GPUs use SIMT (Single Instruction, Multiple Threads): thousands of threads executing the same instruction on different data. TPUs use systolic data flow: values streaming through a fixed grid. SIMT handles irregular workloads gracefully; systolic arrays maximize efficiency for regular workloads.

2. Memory architecture — GPUs use complex cache hierarchies to hide unpredictable memory latency. TPUs use large on-chip buffers with predictable access patterns, eliminating caching overhead entirely.

3. Programmability — GPUs let you write custom CUDA kernels with fine-grained control. TPUs require compilation through XLA, which optimizes automatically but limits what you can express.

4. Ecosystem — CUDA has 15+ years of libraries, tools, and community knowledge. TPU tooling is younger and entirely controlled by Google.

Neither approach is universally better. They optimize for different assumptions about your workload.

Programming TPUs: The XLA Model

Since you can’t program TPUs directly, understanding XLA is essential.

XLA captures your computation as a graph of operations: matrix multiplies, convolutions, activations, and so on. It then optimizes this graph by fusing operations together, eliminating redundant computation, and arranging memory layout for efficient access. Finally, it generates TPU instructions that map onto the systolic array.

The trade-off: XLA handles optimization automatically, but you lose the ability to hand-tune performance. If an operation compiles poorly, your options are limited compared to CUDA, where you can always write a custom kernel.

This is why TPUs work best with standard architectures. Transformers, CNNs, and common layer types compile efficiently. Custom operations or unusual control flow can hit XLA limitations.

Practical Considerations: Batch Size and Memory

TPUs have a simpler memory hierarchy than GPUs: High Bandwidth Memory (HBM) connects to on-chip buffers, which feed the systolic array. The key constraint is keeping the array fed with data.

This is why batch size matters enormously on TPUs. Larger batches mean more data reuse in the systolic array, better amortization of memory transfer overhead, and higher utilization of compute units. If your batch size is too small, the array spends cycles waiting for data instead of computing.

This differs from GPUs, where the flexible threading model handles small batches more gracefully. On TPUs, you often need to redesign your training pipeline around larger batches to achieve good performance.

When TPUs Struggle

The systolic array’s efficiency comes from predictable, dense, regular computation. When workloads deviate from this pattern, performance degrades.

Sparse operations are the clearest example. If your matrices are mostly zeros, the systolic array still processes every zero. There’s no sparse matrix acceleration. Sparse attention mechanisms, mixture-of-experts with dynamic routing, or sparse activations can significantly underutilize TPU hardware.

Dynamic shapes cause problems because XLA compiles for specific tensor dimensions. Variable sequence lengths or dynamic batching require either padding (wasting compute) or recompilation (adding latency).

Complex control flow maps poorly to the systolic array’s fixed data flow. Most neural networks are straight-line computation, but models with significant branching can struggle.

Scaling: TPU Pods

Individual TPUs connect into pods through high-speed custom interconnects. Unlike GPU clusters that use standard InfiniBand or Ethernet, TPU pods use a 2D or 3D torus topology where each chip communicates directly with its neighbors.

This topology is optimized for the communication patterns of distributed ML training, particularly the all-reduce operations used in gradient synchronization. The interconnect is tightly coupled with the TPU architecture, rather than being a separate networking layer you configure independently.

This tight coupling means TPU pods can be extremely efficient for workloads that fit their communication patterns, but less flexible if your distributed training approach differs from Google’s assumptions.

Making the Choice

Given everything above, here’s when each option makes sense:

Choose GPUs when:

• You need framework flexibility, especially PyTorch-first workflows

• Your workloads include sparse operations or dynamic shapes

• You want multi-cloud or on-premises deployment

• You need fine-grained control over kernel implementations

• You’re prototyping and need fast iteration

Choose TPUs when:

• You’re running large-scale training or inference on dense, regular models

• You’re using TensorFlow or JAX and standard architectures

• You can use large batch sizes that saturate the systolic array

• Latency predictability matters (TPUs don’t thermal throttle like GPUs)

• You’re committed to Google Cloud infrastructure

The Broader Lesson

TPUs embody a bet: that ML workloads are important and stable enough to justify specialized silicon. General-purpose hardware can run anything but excels at nothing. Specialized hardware excels at specific workloads but becomes useless when requirements change.

So far, Google’s bet has paid off. The workloads that matter most, transformers and large language models, are exactly the dense matrix operations that systolic arrays handle best.

The lesson for engineers isn’t that TPUs are better or worse than GPUs. It’s that hardware architecture embodies assumptions about workloads. When you understand those assumptions, you can match your problem to the right hardware. When you don’t, you end up fighting the architecture instead of leveraging it.

Subscribe now

Dive deeper into ML System Design @ System Overflow

If you’ve worked in distributed systems or backend infrastructure, you’ve almost certainly come across Jeff Dean and Sanjay Ghemawat. Together, they co-created MapReduce and Bigtable, two foundational systems that shaped modern large-scale data processing. Jeff Dean later went on to co-author Spanner and lead major efforts behind TensorFlow, extending that lineage into globally consistent databases and large-scale machine learning.

Recently, they published a comprehensive guide on performance optimization, and it’s a goldmine. Let us extract the core principles that can transform how you think about performance.

The 3% That Matters

Knuth famously said “premature optimization is the root of all evil.” But here’s the full quote most people miss:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.“

The key insight? If you ignore performance entirely during development, you’ll end up with a flat profile—performance lost everywhere, no obvious hotspots. Much harder to fix later.

Why “Fix It Later” Fails

1. Flat profiles are hell to optimize - No clear starting point when slowness is distributed everywhere

2. Library users can’t fix your mess - They don’t understand your internals well enough

3. Heavy-use systems resist change - Hard to refactor production code serving millions of requests

4. Overprovisioning masks problems - You throw hardware at issues that could’ve been prevented with better code

The golden rule: When writing code, choose the faster alternative if it doesn’t significantly hurt readability.

The Art of Estimation

Before diving into optimization, develop intuition for what matters. Ask yourself:

• Is it test code? → Focus on asymptotic complexity only

• Is it application-specific? → Identify hot paths vs. initialization code

• Is it library code? → Assume it’ll be used in performance-critical contexts

Back-of-the-Envelope Math Still Works

Here’s the latency table every engineer should memorize (updated for 2025):

Share

Example: Quicksort a Billion Numbers

Let’s estimate sorting 1 billion 4-byte integers:

1. Memory bandwidth: 4GB × 30 passes ÷ 16GB/s = 7.5 seconds

2. Branch mispredictions: 30B comparisons × 50% mispredicted × 5ns = 75 seconds

3. Total: ~82.5 seconds (branch mispredictions dominate!)

Example: Generate Web Page with 30 Thumbnails

Serial reads from disk:

• 30 images × (5ms seek + 10ms transfer) = 450ms

Parallel reads across K disks:

• Same work, latency drops by factor of K = ~15ms (with hundreds of disks)

Serial reads from SSD:

• 30 images × (20µs + 1ms) = ~30ms

This math takes 30 seconds but saves hours of implementation time.

Measurement: Your #1 Tool

Profiling Strategies

Start with pprof for high-level CPU profiling. Move to perf for hardware counter details.

Critical practices:

• Build with optimizations + debug symbols

• Write microbenchmarks for iteration speed

• Emit performance counter readings for precision

• Profile lock contention separately (can hide CPU bottlenecks)

When Profiles Are Flat

No obvious hotspot? Try this:

1. Many small wins compound - Twenty 1% improvements = 20% total gain

2. Look at loop call stacks - Restructure from the top down

3. Replace generality with specialization - Custom code beats generic libraries

4. Reduce allocations - Get heap profiles, target allocation count

5. Use hardware counters - Cache miss rates reveal hidden costs

API Design for Performance

Bulk Operations

Single-item APIs force expensive boundary crossings. Add bulk variants:

// Before
util::StatusOr<LiveTensor> Lookup(const TensorIdProto& id);

// After - 1000x less overhead
struct LookupKey {
  ClientHandle client;
  uint64 local_id;
};
bool LookupMany(absl::Span<const LookupKey> keys,
                absl::Span<tensorflow::Tensor> tensors);

Real impact: Reduced per-call overhead from milliseconds to microseconds.

View Types Over Copies

// Slow - forces copies
void ProcessData(const std::vector<int>& data);

// Fast - caller chooses container
void ProcessData(absl::Span<const int> data);

This lets callers use std::vector, absl::InlinedVector, arrays, or anything contiguous.

Thread-Compatible > Thread-Safe

Default to external synchronization. Internal locks are wasted overhead when callers already synchronize:

// Before - internal lock
TransferPhase HitlessTransferPhase::get() const {
  MonitoredMutexLock l(&mutex_);
  return phase_;
}

// After - caller synchronizes
TransferPhase HitlessTransferPhase::get() const { 
  return phase_; 
}

Result: 43 seconds → 2 seconds in production workload.

Algorithmic Wins

These are rare but devastating when found.

Example: O(N²) → O(N)

Before: Adding graph nodes/edges one at a time to cycle detector

After: Add entire graph in reverse post-order

Result: Cycle detection becomes trivial.

Example: Replace Sorted Intersection with Hash Table

// Before: O(N log N)
std::set_intersection(sources1.begin(), sources1.end(),
                      sources2.begin(), sources2.end());

// After: O(N)
absl::flat_hash_set<Node*> sources_set(sources1.begin(), sources1.end());
for (Node* src : sources2) {
  if (sources_set.contains(src)) { /* found common source */ }
}

Impact: 28.5s → 22.4s (21% improvement) on large compilations.

Memory Optimization

Compact Representations

Every byte matters at scale. Consider:

// Bad - 64 bits wasted per pointer on modern machines
std::vector<Node*> graph;

// Better - 32-bit indices if <4B nodes
std::vector<uint32_t> node_indices;
Node nodes[];  // Contiguous allocation

Benefits:

• Smaller memory footprint

• Better cache locality

• Less allocator overhead

Inlined Storage for Small Collections

// Before - always heap allocates
std::vector<int> small_list;

// After - stack allocation for ≤N elements
absl::InlinedVector<int, 8> small_list;

No allocation overhead when size ≤ 8.

Arrays Instead of Maps

// Before
gtl::flat_map<int, int> payload_type_to_frequency;

// After - payload types are 0-127
struct PayloadTypeMap {
  int map[128];
};

Benchmark improvement: 26-31% faster access.

Bit Vectors for Dense Integer Sets

// Before
dense_hash_set<ZoneId> zones;  // Heavy allocation

// After
util::bitmap::InlinedBitVector<256> zones;  // Single allocation

bool ContainsZone(ZoneId zone) const {
  return zone < zones.size() && zones.get_bit(zone);
}

Results: 26-31% faster in real workloads.

Reducing Allocations

Every allocation has three costs:

1. Allocator CPU time

2. Initialization overhead

3. Cache line fragmentation

Avoid Needless Allocations

// Before - allocates every time
std::shared_ptr<DeviceInfo> dinfo = 
    std::make_shared<DeviceInfo>();

// After - reuse static instance
static const std::shared_ptr<DeviceInfo>& empty_device_info() {
  static auto* result = new std::shared_ptr<DeviceInfo>(
      std::make_shared<DeviceInfo>());
  return *result;
}

21% throughput increase in production.

Reuse Temporaries

// Before - reallocates every iteration
for (const auto& item : items) {
  ResourceRecord record;  // ← Expensive!
  ProcessRecord(record);
}

// After - reuse allocation
ResourceRecord record;
for (const auto& item : items) {
  record.Clear();
  ProcessRecord(record);
}

Caveat: Reset periodically to prevent unbounded growth.

Reserve Container Capacity

// Before - multiple reallocations
std::vector<int> results;
for (int i = 0; i < n; ++i) {
  results.push_back(compute(i));
}

// After - single allocation
std::vector<int> results;
results.reserve(n);
for (int i = 0; i < n; ++i) {
  results.push_back(compute(i));
}

Avoiding Unnecessary Work

Fast Paths for Common Cases

Handle ASCII in UTF-8 parsing:

// Fast path - process 8 ASCII bytes at once
while ((src_limit - src >= 8) &&
       (((UNALIGNED_LOAD32(src) | UNALIGNED_LOAD32(src+4)) 
         & 0x80808080) == 0)) {
  src += 8;
}

// Handle trailing ASCII bytes
while (src < src_limit && Is7BitAscii(*src)) {
  src++;
}

// Fall back to state machine for non-ASCII
if (src < src_limit) {
  UTF8GenericScan(/*...*/);
}

Precompute Expensive Values

// Before - computed in every hot loop iteration
bool kernel_is_expensive = kernel->IsExpensive();
bool is_merge = IsMerge(node);
bool is_enter = IsEnter(node);

// After - computed once during initialization
struct NodeItem {
  bool kernel_is_expensive : 1;
  bool is_merge : 1;
  bool is_enter : 1;
  bool is_enter_exit_or_next_iter : 1;
};

// Hot path check
if (!item->is_enter_exit_or_next_iter) {
  // Fast path - no special handling needed
}

Defer Until Needed

// Before - always compute
HloSharding alt = user.sharding().GetSubSharding(/*...*/);
if (condition) { use(alt); }

// After - compute only when needed
if (condition) {
  HloSharding alt = user.sharding().GetSubSharding(/*...*/);
  use(alt);
}

Saved 43 seconds → 2 seconds by deferring expensive call.

Specialize Generic Code

Replace regex with simple operations when possible:

// Before
if (RE2::FullMatch(token, "prefix.*")) { /*...*/ }

// After
if (absl::StartsWith(token, "prefix")) { /*...*/ }

Or custom implementations for critical paths:

// Before - sprintf for IP address
StringPrintf("[%s]:%d", ip.ToString().c_str(), port);

// After - direct formatting
StrCat(a1, ".", a2, ".", a3, ".", a4, ":", port);

4x faster in monitoring hot paths.

Caching

// Cache based on fingerprint
uint64 fp = Fingerprint(proto);
{
  absl::MutexLock l(&cache_mu);
  auto it = cache.find(fp);
  if (it != cache.end()) {
    return it->second;  // Cache hit
  }
  // Parse and cache
  auto result = ParseProto(proto);
  cache[fp] = result;
  return result;
}

Help the Compiler

Compilers are smart but conservative. You can help:

Use Raw Pointers in Hot Loops

// Before - absl::Span has overhead
ForEachState(const Shape& s, 
             absl::Span<const int64_t> base,
             absl::Span<const int64_t> count);

// After - raw pointers are faster
ForEachState(const Shape& s,
             const int64_t* const base,
             const int64_t* const count);

Hand-Unroll Critical Loops

// Before
while ((p + 4) <= e) {
  STEP;
}

// After - 16 bytes at a time
while ((e - p) >= 16) {
  STEP; STEP; STEP; STEP;
}
while ((p + 4) <= e) {
  STEP;
}

Replace FATAL with DCHECK

// Before - forces frame setup
default:
  ABSL_LOG(FATAL) << "Invalid tag";
  return sizeof(DynamicNode);

// After - compiles away in optimized builds
default:
  ABSL_DCHECK(false) << "Invalid tag";
  return sizeof(DynamicNode);

Eliminates frame setup costs in release builds.

Reduce Stats Collection

Balance utility vs. cost:

Sample, Don’t Measure Everything

// Before - update 39 histograms per request
UpdateHistograms(request);

// After - sample 1 in 32 requests
if (request_count % 32 == 0) {
  UpdateHistograms(request);
}

Google Meet used this during COVID surge to handle traffic spikes.

Drop Unused Stats

// Removed expensive alarm/closure counting
// that nobody looked at
// Result: 771ns → 271ns per alarm

Precompute Logging Decisions

// Before - checked every iteration
for (int i = 0; i < 1000000; ++i) {
  if (VLOG_IS_ON(3)) {
    VLOG(3) << "Processing " << i;
  }
}

// After - check once
const bool vlog_3 = VLOG_IS_ON(3);
for (int i = 0; i < 1000000; ++i) {
  if (vlog_3) {
    VLOG(3) << "Processing " << i;
  }
}

Results: 8-10% improvement on compute-heavy loops.

When to Optimize

Not all code deserves the same scrutiny:

• Test code: Optimize asymptotic complexity only

• Application code: Focus on request handling, not initialization

• Library code: Assume worst case—it’ll end up on hot paths

• Infrastructure code: Every nanosecond multiplies across thousands of servers

Code Size Matters Too

Large binaries mean:

• Longer compile/link times

• More memory pressure

• Worse instruction cache behavior

• Harder deployment

Sometimes smaller, simpler code is faster despite being “less optimized.”

The Meta-Lesson

Jeff and Sanjay’s guide isn’t about premature optimization. It’s about informed decision-making:

1. Understand the cost model of your system

2. Make the fast choice when it’s equally simple

3. Measure before complex optimizations

4. Think in orders of magnitude, not percentages

5. Compound small wins into big gains

Most importantly: Don’t wait for performance problems to think about performance. Build it right from the start, and save yourself from the architectural debt that’s nearly impossible to pay off later.

Want more deep dives on systems engineering? Follow System Overflow for weekly breakdowns of distributed systems, databases, and infrastructure engineering.

Further Reading:

• Original Abseil Performance Guide

• Jeff Dean’s Numbers Everyone Should Know

• Google pprof Profiler

This evolution is less about optimization, and more about knowing when an architecture has hit its ceiling.

The Core Problem (That Never Changes)

Given millions of items, how do you return the most relevant ones for a user in ~100ms at global scale?

Key constraints:

• Freshness: React to recent user behavior

• Latency: Stay within tight p99 budgets

• Cost: Avoid computing and storing things no one sees

• Model complexity: Support increasingly powerful ML models

Every architecture answers these constraints differently.

The Four Phases of Recommendation Architecture

Phase 1: Offline Batch Scoring (2008–2012)

In the earliest phase, LinkedIn relied on offline batch jobs that precomputed recommendation scores and stored them for lookup.

This approach failed quietly. Storage requirements exploded as users and items grew, and recommendations became stale almost immediately. Profile updates or job changes took hours or days to appear.

Why this matters?

Offline systems don’t usually break. They decay. Relevance fades slowly, and by the time metrics move, the architecture is already holding you back.

Phase 2: Nearline Scoring (2012–2015)

To reduce staleness, LinkedIn introduced nearline recomputation triggered by user actions.

Freshness improved, but consistency didn’t. Some recommendations were updated quickly while others lagged behind. Distributed triggers introduced coordination complexity and partial failures that were hard to observe.

Why this matters?

Hybrid systems often look like progress, but they introduce uneven behavior that’s harder to debug than full batch or full real-time systems.

Phase 3: Online Scoring on CPUs (2015–2020)

The next shift moved scoring into the request path. Recommendations were generated and scored in real time when a user loaded a page.

Freshness was solved, but latency became the dominant constraint. With a strict end-to-end budget, models had to be simple enough to run on CPUs. Feature richness and model depth were limited, and cold-start problems persisted.

Why this matters?

Online scoring fixes staleness, but it puts a ceiling on intelligence. Latency budgets become product constraints.

Phase 4: Decoupled Architecture with Remote GPU Scoring (2020–Present)

The current system separates candidate generation from scoring and offloads inference to remote GPU services.

This architectural split changed what was possible. Candidate generation focused on recall, while scoring models could grow far more sophisticated. Embedding-based retrieval enabled semantic matching, reducing cold-start issues.

Why this matters?

GPUs didn’t just make models faster—they made entirely new classes of models possible.

What Happens in a Single Request

When a user opens LinkedIn Jobs, the system generates candidates from multiple sources, narrows them using embedding similarity, and ranks them using GPU-powered models. Hundreds of features and real-time context are evaluated, all within roughly 100 milliseconds at the 99th percentile.

Decoupling recall from ranking allows each stage to evolve independently without blowing the latency budget.

Cost and Failure Modes

Despite GPU infrastructure being expensive, overall system cost dropped dramatically. The system stopped precomputing and storing scores that were never viewed and shifted to on-demand inference.

Failure modes shifted as well. Offline systems fail slowly through staleness. Online systems fail instantly through latency. GPU-based systems fail operationally through serving and orchestration complexity.

Every architectural upgrade trades one kind of risk for another. Maturity isn’t eliminating failures, it’s choosing the ones you can handle.

Choosing the Right Architecture

This evolution isn’t linear progress where the final stage is always best.

Offline batch systems still make sense at small scale. Nearline or online scoring becomes necessary when staleness hurts engagement. GPU-based inference only pays off when traffic, latency pressure, and business value align.

Premature architectural ambition is just another form of technical debt.

The Real Takeaway

LinkedIn’s biggest gains didn’t come from tuning models, but they came from recognizing architectural limits and crossing them decisively.

If your system feels stuck despite constant optimization, the problem is probably architectural, not algorithmic.

Share

Learn more about ML design concepts on System Overflow.

Design a web service that allows users to store and share text snippets through unique URLs. Users can paste text content, receive a shareable link, and optionally set expiration times. The system should handle basic text storage, retrieval, and URL generation similar to Pastebin or GitHub Gist.

Share

Every request you make at Airbnb - from searching for a stay to loading support data - eventually hits Mussel, Airbnb’s multi-tenant key-value store. At normal load it serves millions of reads smoothly, but during spikes, bulk uploads, or bot traffic, it has to stay fast without taking the rest of the platform down.

Airbnb’s first solution used simple per-client QPS limits. That was enough to prevent total meltdowns, but not enough to maximize useful work while staying within capacity. The real challenge was building adaptive traffic management that reacts to changing workloads in real time.

In this article, you will see how Airbnb evolved Mussel from static rate limiting to a layered QoS system, and how you can apply the same patterns - resource-aware limits, local feedback control, and hot-key protection - in your own systems.

The Challenge

Mussel sits in front of a storage engine as a fleet of stateless dispatcher pods running on Kubernetes. Every online product area at Airbnb sends traffic to it: point lookups, range scans, bulk writes, and internal tools. On top of normal diurnal cycles, Mussel also sees:

- Sudden user spikes, for example when a listing goes viral
- Large analytical or backfill jobs that scan huge ranges
- Misbehaving crawlers or outright DDoS-like bursts

The original design used a Redis-backed rate limiter: each caller had a static per-minute quota, and dispatchers incremented a counter per request. If you exceeded your quota, Mussel returned HTTP 429. This was simple and gave caller-level isolation: one bad client could not take the entire system down.

Over time, two deeper problems appeared.

1. Cost variance between requests

A one-row read and a 100,000-row range scan both counted as “1 request”. In reality they could differ by orders of magnitude in:

- CPU and memory usage
- Network bytes
- Disk I/O and cache pressure

This meant you could stay within your QPS but still quietly crush the backend with a few very expensive queries. The system had no concept of real resource cost per request.

2. Traffic skew and hot spots

Rate limits were per caller, not per data item. Imagine a popular listing that appears on a front page. Thousands of different callers all read the same key. Each caller respects its quota, but all these reads pound a single shard. That shard slows down, and now even unrelated keys hosted there get slow. Isolation at the caller level did not give isolation at the data or shard level.

A useful analogy is a supermarket with a “10 items per customer” rule but no control on how many people swarm one shelf. You avoid one person hogging the checkout, but if 500 people all grab from the same shelf at once, that aisle still collapses.

Common solutions like global QPS caps or static per-service quotas fail here because they:

- Do not distinguish cheap from expensive work
- Do not respond quickly enough to micro-bursts and hot keys
- Do not express priority across different types of traffic

This is a common challenge whenever you run a multi-tenant backend that serves heterogeneous workloads: databases, search clusters, caches, or shared internal platforms. You are not just limiting how many calls arrive; you are managing which work gets done under constrained resources.

Airbnb’s Solution Architecture

Airbnb’s engineers redesigned Mussel’s QoS as a layered control system rather than a single global rate limit. The layers are:

1. Resource-aware rate control (RARC) using request units
2. Load shedding based on latency feedback and request criticality
3. Hot-key detection and mitigation to protect shards from skewed access

You can read this as: “price requests correctly, then adapt to stress, then neutralize amplification patterns.” Here is how each layer works and how it generalizes.

1. Resource-aware rate control with request units

Instead of counting raw QPS, Mussel now charges each operation in request units (RU). An RU is a synthetic cost metric that reflects:

- A base cost per call
- Data volume (rows or bytes)
- Observed latency for that call

You can think of this as a linear pricing model:

> RU = base_cost + weight_bytes bytes + weight_latency latency_ms

Weights are calibrated from load tests that roughly balance CPU, network, and disk cost. The exact coefficients are not the important part. The important idea is that expensive operations consume more of a client’s quota than cheap ones, using metrics you can easily observe at the proxy.

Each caller now has a static RU quota per time window, not a raw QPS quota. Each dispatcher maintains a local token bucket: on each request, it computes the RU cost and decrements tokens. When tokens are gone, Mussel rejects the request with a throttling code.

Airbnb chose this model because it preserved a simple, static contract to clients (you get N RU per minute), while internally allowing much better fairness across workloads. This pattern - “convert requests into resource units, then rate limit on units” - is useful any time you have:

- A shared backend with variable-cost operations
- The ability to measure simple per-request features

Subscribe now

2. Load shedding with local latency feedback

RU-based quotas smooth average behavior, but they do not react instantly to rapid changes. Airbnb added a load-shedding layer that uses two local signals inside each dispatcher:

- A latency ratio: short-term p95 latency divided by long-term p95 latency
- A queue delay threshold: based on how long requests spend waiting in the dispatcher thread pool

The latency ratio is a compact way to detect when the system is getting slower. A ratio near 1 means “steady.” If the short-term p95 grows sharply, the ratio drops toward 0.3 or lower. When it crosses a threshold, the dispatcher concludes “we are entering overload” and starts to penalize less critical traffic classes by artificially inflating their RU cost or dropping queued requests earlier.

The queue control is inspired by CoDel-style algorithms that do not just look at queue length, but at sojourn time (how long each request sat in the queue). If that wait time exceeds a target for long enough, the dispatcher starts failing new arrivals quickly instead of letting them pile up and time out.

Airbnb also tags traffic with criticality tiers (for example, user-facing vs batch). Under stress, lower tiers back off first, preserving headroom for critical paths like customer support or trust and safety.

This pattern of local, feedback-based load shedding is powerful when you need to:

- React within milliseconds to backend slowdowns
- Protect high-priority workloads without human intervention
- Keep the control logic independent per proxy instance

3. Hot-key detection and DDoS mitigation

Even with good rate limits, a “stampede” of reads for one record can overload its shard. Airbnb addressed this with a three-part hot-key defence at each dispatcher:

1. Real-time top-k tracking of keys using a constant-space streaming algorithm
2. Short-lived local caching of hot keys
3. Request coalescing for in-flight reads of the same hot key

Every incoming key updates a compact data structure that approximates the most frequent keys. When a key crosses a hotness threshold, the dispatcher starts serving it from a small process-local LRU cache with a very short TTL (on the order of seconds). This is enough to ride out a news spike or a bot burst without requiring a global cache.

If multiple requests for a hot key arrive while a cache miss is in progress, the dispatcher records that there is an in-flight request and attaches new callers to the same future. When the backend response arrives, it fans out to all waiters. In practice this means that for each dispatcher pod, at most one in-flight backend request per hot key is active at a time.

This general pattern - “detect hot keys, cache locally with tiny TTLs, coalesce in-flight requests” - is broadly applicable to any key-based storage or cache. It converts N identical expensive reads into 1 read plus N cheap local responses.

Trade-offs & Considerations

Airbnb’s approach trades simplicity for control and resilience.

On the positive side, you get:

- Finer-grained fairness: heavy range scans cannot starve cheap point reads
- Better protection of critical paths during overloads
- Robustness to traffic skew and DDoS-style patterns

On the cost side, you accept:

- More moving parts: RU accounting, latency feedback, queue control, top-k tracking, local caches
- Tuning complexity: choosing RU weights, latency thresholds, and hot thresholds
- Some approximation error: RU formulas and streaming algorithms are not perfect

This architecture works best when:

- You own a clear proxy or gateway where all traffic passes
- You can cheaply measure basic per-request metrics (bytes, latency)
- You have heterogeneous workloads and multi-tenant usage

It may be overkill for a small service with uniform traffic where simple per-client QPS limits suffice, or for systems where you cannot easily change client quotas or introduce admission control.

You also need to think about failure modes: what happens if the central quota store (like Redis) is down, or if latency metrics get corrupted, or if hot-key detection misfires. Airbnb’s design keeps most control loops local to each dispatcher, which reduces blast radius: if one node’s control logic behaves badly, it does not directly affect others.

Conclusion

Airbnb’s evolution of Mussel from static QPS limits to adaptive traffic management illustrates a powerful system design pattern for multi-tenant backends.

You start by measuring work in resource units, not requests. You then layer fast, local feedback loops to shed load based on latency and queue health. Finally, you neutralize amplification patterns like hot keys with detection, caching, and request coalescing.

If you run a shared data service, cache, or API that different teams depend on, these ideas map directly to classic system design concepts: token buckets and RU-based rate limiting, admission control, backpressure, prioritized QoS, and hotspot mitigation. As your traffic and tenants grow, this style of architecture can be the difference between “we stayed up” and “everything was technically within quota but still fell over.”

The Core Concept

A message queue is a durable buffer that sits between services, letting them communicate asynchronously instead of waiting for each other. When your application needs to do work, it writes a message to the queue and immediately continues. Separate consumer services read messages from the queue and process them at their own pace.

Think of a message queue like a restaurant’s order ticket system. When you place an order, the waiter writes it down and immediately returns to serve other customers. The kitchen processes orders as fast as they can, but diners aren’t standing at the counter waiting. If the kitchen gets slammed with orders, tickets stack up, but no customer is told “we’re too busy right now, come back later.”

This architectural shift changes everything about how systems handle load. Imagine your e-commerce API receives 10,000 order requests per second during a flash sale, but your payment processor can only handle 2,000 per second. Without a queue, 8,000 requests would fail or timeout. With a queue, all 10,000 requests succeed immediately (the message is queued), and the payment processor works through the backlog over the next few minutes.

The trade-off is clear: you give up instant confirmation in exchange for resilience. Users don’t know immediately if their payment succeeded, they get a “we’re processing your order” message instead. This works perfectly for orders, background jobs, and notifications, but poorly for interactive requests like search where users need immediate results.

How This Works in Production

Let’s continue with the e-commerce example and see how Amazon SQS (a popular message queue service) handles this in production. When a customer clicks “Place Order,” your API writes the order details to an SQS queue in under 100 milliseconds and returns a confirmation page. The customer sees “Order received” and can continue shopping.

Behind the scenes, multiple consumer instances are continuously polling the queue for work. When a consumer receives a message, SQS marks it invisible to other consumers for 30 seconds (the visibility timeout). This prevents two consumers from processing the same order simultaneously. The consumer validates the order, charges the payment method, updates inventory, and sends a confirmation email. Once complete, it acknowledges the message, and SQS permanently deletes it.

Here’s the counterintuitive part: most production queues use “at least once” delivery, meaning the same order might be processed twice if a consumer crashes before acknowledging. This seems like a bug, but it’s actually the practical choice. True “exactly once” processing requires complex distributed transactions that add significant latency. Instead, systems make their processing idempotent: they store processed order IDs in a cache for 24 to 72 hours. If a duplicate arrives, they check the cache and skip reprocessing. This achieves the same correctness with much simpler infrastructure.

During normal load, messages wait in the queue for only seconds. During that flash sale spike, the queue might accumulate thousands of messages, and end-to-end processing time stretches to minutes. But no orders are lost, and the system never tells customers it’s overloaded. You simply add more consumer instances to work through the backlog faster.

Subscribe now

Key Takeaway

Message queues transform how distributed systems handle load by decoupling services and absorbing traffic spikes, trading immediate feedback for resilience and independent scaling. Understanding Message Queue Fundamentals is foundational for building scalable systems. Learn more in-depth about Message Queue Fundamentals on System Overflow, with detailed cards covering advanced patterns, edge cases, and production scenarios.

Learn more in-depth about Message Queue Fundamentals on System Overflow

Last Day: Cyber Monday Sale - 15% OFF

Load shedding is the practice of intentionally rejecting some requests when a system approaches capacity limits, protecting it from complete collapse. Think of it like a nightclub with a maximum occupancy: instead of letting everyone in and creating a dangerous, unpleasant experience for all, the bouncer controls entry to maintain a good experience for those inside.

The fundamental challenge Netflix engineers identified is that traditional autoscaling assumes you have time to respond. When a popular show drops at midnight and millions of users flood the service within minutes, spinning up new server capacity takes too long. The alternative, provisioning enough capacity for theoretical maximum peaks, would mean running servers at perhaps 20% utilization most of the time, wasting millions of dollars annually.

This is where Netflix’s conceptual resilience model becomes critical. They quantify system health using two buffers. The Success Buffer represents how much traffic above baseline a service can handle without performance degradation. If your service normally runs at 1,000 requests per second and can handle up to 1,500 requests per second before latency increases, you have a 500 request per second Success Buffer.

The Failure Buffer is equally important but often overlooked. This represents the system’s capacity to gracefully reject excess requests without cascading failures. When traffic hits 1,600 requests per second, the system needs enough headroom to evaluate incoming requests, make rejection decisions, and send proper error responses. Without this buffer, the service simply freezes or crashes, taking everything down.

The breakthrough Netflix achieved was recognizing that not all requests deserve equal treatment during overload. Traditional load shedding dropped requests randomly, like closing your eyes and randomly turning away people at the nightclub entrance. This means a paying customer trying to enter might get rejected while someone just looking for the bathroom gets in.

Service-Level-Prioritized Load Shedding assigns priorities to different request types based on their business value. When you click play on a show, that’s a high-priority, user-initiated playback request. But Netflix also makes prefetch requests in the background, preloading content you might watch next. During normal operation, prefetching improves experience. During overload, these low-priority requests get shed first, preserving capacity for actual playback.

How Netflix Implements Prioritized Load Shedding

The implementation involves moving load shedding decisions from Netflix’s centralized API Gateway down to individual microservices. This architectural choice creates a surprising advantage: it allows critical requests to dynamically steal capacity from non-critical operations within the same application instance.

Consider a specific scenario during a major content launch. A backend service handling both user playback requests and background analytics processing normally allocates resources equally. Under traditional load shedding at the gateway, if the gateway drops 30% of traffic randomly, both critical playback and non-critical analytics get reduced proportionally. The service itself still wastes precious CPU cycles on analytics while users can’t watch their shows.

With service-level prioritization, the picture changes dramatically. As CPU utilization climbs to 60%, the service automatically begins shedding background analytics requests while continuing to process all playback requests. At 70% utilization, it might shed prefetch operations. Only when utilization reaches 80% does it start rejecting even some user-initiated requests, and even then, it prioritizes based on factors like whether this is a new session versus an ongoing stream.

The counterintuitive insight here is that optimal load shedding actually increases total request rejection rates compared to waiting until the last moment. By proactively shedding low-priority traffic at 60% utilization, the system maintains its Success Buffer for high-value requests. Services that wait until 95% utilization to start shedding often experience cascading failures because they’ve exhausted both their Success and Failure Buffers simultaneously.

Netflix automated this across hundreds of microservices through three integrated pillars. Priority assignment happens early in the request lifecycle through headers that propagate downstream. Critically, services can only maintain or decrease priority, never escalate it, preventing gaming of the system. A prefetch request tagged as low-priority stays low-priority throughout its journey.

Central configuration generates unique load-shedding functions for each service cluster based on its specific utilization metrics. One service might use CPU as its primary signal, starting non-critical shedding at 60% CPU. Another might use request queue depth, beginning shedding when 200 requests are queued. These thresholds map utilization levels and request priorities to specific rejection probabilities. At 65% CPU, the system might reject 50% of low-priority requests but 0% of high-priority ones. At 75% CPU, it might reject 100% of low-priority and 20% of high-priority requests.

Automated validation through Netflix’s Chaos Automation Platform ensures every cluster has adequate buffers before major launches. Engineers inject artificial load spikes weeks in advance, measuring whether services maintain stability and shed appropriately. This validation caught cases where services would have failed during actual launches, allowing teams to adjust configurations proactively.

The retry strategy prevents the thundering herd problem where shed requests immediately retry, amplifying the overload. When server-side shedding activates, clients scale back all retries. Under heavy load, only high-priority requests retry, and even those use exponential backoff. This coordination between client and server behavior is crucial. Without it, shedding 30% of requests at the server just triggers 30% more retries from clients, creating a feedback loop that makes overload worse.

Trade-offs and Architectural Decisions

Implementing service-level prioritized load shedding introduces significant complexity compared to simple gateway-based approaches. Each service team must instrument their code to understand and propagate priorities, instrument utilization metrics, and test shedding behavior. For organizations with dozens rather than hundreds of services, centralized gateway shedding might provide 80% of the benefit with 20% of the complexity.

The decision of when to start shedding involves balancing user experience against resource efficiency. Starting shedding too early (say, at 40% utilization) wastes capacity and unnecessarily degrades service for low-priority but still valuable requests. Starting too late (at 90% utilization) risks exhausting your Failure Buffer and experiencing cascading failures. Netflix found the sweet spot typically falls between 60-70% utilization for non-critical shedding.

Different companies make different architectural choices based on their traffic patterns. Services with gradual, predictable load increases might rely primarily on autoscaling with minimal load shedding. Services with sudden spikes (ticket sales, limited product drops) need aggressive load shedding. Companies like Ticketmaster implement virtual waiting rooms rather than shedding, queuing excess users transparently. The right approach depends on whether maintaining queue position has business value.

Cost implications are substantial but nuanced. Load shedding allows running closer to capacity limits, potentially reducing infrastructure costs by 30-40%. However, the engineering investment in building, testing, and maintaining the shedding infrastructure is significant. Organizations must weigh these ongoing engineering costs against infrastructure savings and improved reliability during peak events.

Key Takeaway

Service-level prioritized load shedding transforms graceful degradation from a blunt instrument into a precision tool, ensuring systems protect their most valuable traffic when overwhelmed. By treating requests differently based on business impact, maintaining separate Success and Failure buffers, and automating configuration across distributed systems, services can survive traffic spikes that would otherwise cause complete outages. Learn more about Enhancing Reliability Using Service-Level Prioritized Load Shedding: Netflix at QCon SF 2025 and other system design concepts on System Overflow.

Last Day: Cyber Monday Sale - 15% OFF

Learn more about rate limiting and other system design concepts on System Overflow.

What Are Cache Patterns?

Cache patterns define who controls cache population and when data gets written to your source of truth (your database). Think of caching like a restaurant kitchen: Cache Aside is when the waiter (your application) checks if a dish is ready on the counter, and if not, goes to the kitchen to get it. Read Through is when the waiter always asks the expeditor (cache layer), who handles fetching from the kitchen automatically. Write Back is when orders go on a ticket board first, then get batched to the kitchen later.

Cache Aside puts your application in full control. On reads, your app checks the cache first. On a miss, it loads from the database, stores the result in cache with an expiration time, and returns the data. On writes, you update the database first, then delete the cached entry to prevent serving stale data. This pattern excels for read-heavy workloads where you want fine-grained control over what gets cached.

Read Through moves responsibility to the cache layer itself. Your application always reads through the cache, which automatically fetches from the database on misses. This centralizes logic and enables smart features like request coalescing (combining multiple requests for the same key into one database fetch).

Write Back optimizes for write speed by acknowledging writes immediately to the cache, then persisting to the database asynchronously in batches. This delivers the fastest write latency but risks data loss if the cache fails before flushing.

How This Works in Production

Let’s see how Meta uses Cache Aside with Memcache at massive scale for their social graph data. When you load a friend’s profile, the application first checks Memcache for that user’s data. On a cache hit (which happens over 90% of the time), Memcache returns the data in under a millisecond. This is why profile pages feel instant.

On a cache miss, the application queries MySQL, which takes a few milliseconds for an intra-datacenter round trip. It then stores the result in Memcache with a Time To Live before returning it to you. Here’s the surprising part: Meta deletes cache entries on writes rather than updating them. This prevents a subtle race condition where one server might cache stale data while another is writing new data.

To prevent cache stampedes (where thousands of servers simultaneously miss a hot celebrity’s profile and overwhelm MySQL), Meta uses lease tokens. Only one server gets permission to fetch from the database, while others wait briefly for that result. They also add jitter to expiration times, randomizing TTLs by 10-20%. Without this, synchronized expiry would cause massive synchronized misses.

The trade-off with Cache Aside is application complexity. Your code must handle miss logic, concurrency controls, and invalidation carefully. But this complexity buys you flexibility: you can cache denormalized views, implement different strategies per entity type, and optimize based on specific access patterns.

Key Takeaway

Cache patterns are fundamental architectural decisions that shape your system’s performance characteristics and operational complexity. Cache Aside gives maximum control for read-heavy workloads, Read Through simplifies application code with smart cache infrastructure, and Write Back optimizes write latency with durability trade-offs. Understanding Cache Patterns (Aside, Through, Back) is foundational for building scalable systems. Learn more in-depth about Cache Patterns (Aside, Through, Back) on System Overflow, with 3 detailed cards covering advanced patterns, edge cases, and production scenarios.

Learn more in-depth about Cache Patterns (Aside, Through, Back) on System Overflow

If you have been following along, you know us as Preploop. Today, we’re excited to announce our rebrand to System Overflow.

Why the Change?

The old name suggested we were just another interview prep tool. But that’s never been what we’re about.

We build tools for engineers who want to actually understand system design - not just memorize templates for interviews. Engineers who want depth over breadth. Trade-offs over talking points.

System Overflow better reflects what the platform does: help you systematically master the fundamentals and advanced concepts that make distributed systems work.

Same platform. Same 3000+ cards curated by FAANG+ staff-level experts. Your account, progress, streaks, and entire learning journey remain exactly as they were. Just a better name.

Black Friday: EXTRA 20% Off All Plans

To celebrate the rebrand (and because it’s Black Friday), we’re offering 20% off all pro plans through the end of the week.

Early Adopter Pricing + Black Friday Discount:

• 6 Months Pro: $59 → $47 (20% off)

• 1 Year Pro: $99 → $79 (20% off)

The discount applies automatically at checkout using code BLACKFRIDAY20.

This stacks with our early adopter pricing, which won’t last forever. Once we hit our growth targets, prices go back to regular rates.

What You Get with Pro

• Unlimited learning cards (3000+ cards across System Design aspects ML Systems, and Software Engineering)

• Unlimited design practice problems with AI evaluation

• Company-specific tagging (Google, Meta, Amazon, Netflix, etc.)

• Full access to trade-offs, edge cases, and failure modes

Free tier stays at 6 cards per day if you want to try it first.

Upgrade to Pro - BLACK FRIDAY OFFER

What’s Next

The rebrand is just the beginning. We’re shipping:

• Audio explanations for every card (already live for 100+ concepts)

• YouTube videos breaking down complex system design problems

• AI Mock Interviews (not just another AI gimmick - substantive practice that tests your understanding)

• Company-specific learning paths

If you’ve bookmarked preploop.io, update your links to http://www.systemoverflow.com. All old links redirect automatically, so nothing breaks.

Thanks for being part of this journey.

P.S. Black Friday discount ends November 30th at midnight UTC. Honestly, this is our biggest sale.

The Journey From Send to Response

When you press Send, your message triggers a complex multi-stage process that transforms simple text into contextually aware responses. The system doesn’t just pass your question to a model and return an answer. Instead, it assembles context, validates safety constraints, manages computational resources, and carefully orchestrates the generation process across specialized hardware.

Understanding Context Windows

The core challenge is that language models cannot simply “think” about your question. They process text as sequences of tokens (roughly word fragments), and they have strict limits on how much text they can consider at once. This boundary is called the context window. Modern production models range from 8,000 to over 1 million tokens, but regardless of size, this is a hard constraint. The model literally cannot see beyond it.

Think of context assembly like packing a suitcase with a strict weight limit for a trip. You have system instructions (the essentials you must bring), conversation history (items from previous trips you want to reference), tool definitions (specialized equipment you might need), and your new message (today’s additions). If everything doesn’t fit, the system must decide what to trim or compress. This is why very long conversations eventually “forget” earlier details. The platform summarizes or drops old turns to make room for new ones.

The Two Inference Phases

Once context is assembled and validated through safety checks, the actual inference begins in two distinct phases:

Prefill: The model reads every input token and computes its initial internal state. This is computationally expensive because the model must process the entire context in one pass. For a 2,500 token prompt, this typically takes 400 milliseconds under normal load.

Decode: The model generates output one token at a time, feeding each new token back into its state to produce the next one. This happens at roughly 25 tokens per second, depending on server load and model size.

This is why you see responses stream word by word rather than appearing all at once. The system starts sending tokens as soon as the first one is ready, hiding the remaining computation time behind progressive delivery. For users, this creates the perception of low latency even when generating a full response takes many seconds. Without streaming, you would wait in silence for the complete answer, making the system feel sluggish and unresponsive.

Inside a Production Serving Stack

Let’s walk through a realistic scenario. You open ChatGPT and ask: “What are three unique gift ideas for a software engineer who loves coffee?” with about 2,000 tokens of prior conversation history. The platform assembles roughly 2,500 tokens total including system instructions and tool schemas.

Request Routing and Priority Queuing

The request hits the API gateway, which enforces rate limits and routes based on your subscription tier. As a paid user, your request enters a priority queue that protects you from free-tier traffic spikes. The scheduler groups your request with others targeting the same model version and similar characteristics into a dynamic batch.

Here’s the counterintuitive part: batching actually increases your individual latency slightly (often 50 to 200 milliseconds at the median, sometimes a full second or more at p99 during peak load) but improves overall system efficiency by 2 to 8 times. The platform accepts this trade-off because it dramatically reduces the GPU hours needed to serve millions of requests.

GPU Processing and Inference

Your batched request lands on an NVIDIA H100 GPU, which costs providers roughly to per hour in cloud environments. The prefill phase processes your 2,500 input tokens, taking approximately 400 milliseconds under normal load. This first-token latency dominates your perceived wait time for short prompts. The model then begins decode, generating tokens at roughly 25 per second given current server load and model size.

Caching Optimizations

But the system isn’t just generating text blindly. It’s using two critical optimizations:

KV Caching: Stores intermediate attention computations so the model doesn’t recompute from scratch for every new token. Without this optimization, decode would be prohibitively expensive.

Prompt Caching: Leverages the immutable parts of your context (system instructions and tool schemas), skipping their recomputation entirely across requests. This can reduce prefill time by 60-80% for subsequent requests.

Tool Calls and External Services

After generating about 15 tokens, the model decides it needs current information about coffee accessories. It emits a structured tool call to perform a search. The platform executes this external service call, which adds 800 milliseconds (internal tool latency varies from 100 milliseconds for cached data services to 2-3 seconds for web searches). The results get appended to your context, and the model continues decoding with fresh information. This is why responses sometimes pause briefly mid-generation before continuing with specific details.

Output Safety and Streaming

Throughout decode, output safety checks run in parallel. These add minimal latency (typically 30-80 milliseconds) for most requests but can block content after generation completes if policy violations are detected. The streaming transport uses server-sent events to deliver each token chunk with metadata like token position and status flags, allowing your browser to render progressively.

The full response of roughly 180 tokens completes in about 8 seconds total: 400ms prefill, 800ms tool call, plus decode time. Your browser showed the first word after just 400 milliseconds, making the experience feel nearly instantaneous despite significant backend work.

Trade-offs and Design Decisions

Throughput vs Tail Latency

Production systems face constant tension between throughput and tail latency. Larger batch sizes maximize GPU utilization and reduce cost per request but increase head-of-line blocking. Background workloads like content moderation or summarization can tolerate batches of 64 or 128 requests. Interactive chat with strict latency SLOs (service level objectives, like p95 under 4 seconds to first token) requires smaller batches of 8 to 16 requests or dedicated capacity.

Long Context vs Retrieval Augmented Generation

The choice between long context windows and retrieval augmented generation presents another fundamental trade-off. Google’s Gemini 1.5 advertises a 1 million token window, which eliminates complex retrieval orchestration and reduces tool call latency. However, this increases prefill time proportionally and creates memory pressure. At these scales, providers gate access and prioritize paid tiers to maintain p50 latency under 2 seconds for premium users. Retrieval keeps prompts short and costs low but depends entirely on retrieval quality and introduces external dependencies.

Common Failure Modes

Several failure patterns reveal system boundaries:

Context overflow can truncate critical system instructions, causing the model to ignore constraints or forget available tools.

Tool call loops happen when schemas are ambiguous, with the model repeatedly calling the same function.

Cold starts occur when GPU memory pressure forces worker eviction, adding 15 to 45 seconds of delay while weights reload.

Moderation races waste compute by generating many tokens before output filters block the entire response.

Robust systems cap tool depth, maintain hot GPU pools sized for p95 load, and use early classification to avoid generating content likely to be blocked.

The Big Picture

The architecture of modern LLM serving reveals a critical insight: perceived latency and actual computational cost often point in opposite directions. Streaming, batching, and caching optimize for user experience and infrastructure efficiency simultaneously, but they introduce complexity in error handling, cost attribution, and capacity planning. Systems that master these trade-offs deliver both responsive user experiences and sustainable unit economics. As models grow larger and context windows expand, the engineering challenge shifts from making inference possible to making it economical at scale while maintaining strict latency guarantees for interactive workloads.

Learn more Machine Learning System Design concepts on PrepLoop.io

What Is Leader-Follower Replication?

Leader-follower replication is a distributed data architecture where one node (the leader) coordinates all write operations, while multiple follower nodes copy those changes to provide redundancy and serve read traffic. This design solves a fundamental problem in distributed systems: how do you keep multiple copies of data consistent while handling failures?

The architecture works through a simple but powerful pattern. When a client wants to write data (say, updating a user’s email address), that write goes exclusively to the leader. The leader appends this operation to a replication log with a sequence number, saves it to disk, and then ships this log entry to all followers. Each follower applies these entries in the exact order they were created, ensuring they eventually converge to match the leader’s state.

This single-writer design eliminates write conflicts by construction. Since only one node accepts writes, there’s no ambiguity about which update happened first. Imagine a social media post: the leader decides it’s post number 12,345 in the system, and every follower applies that exact same post with that exact same number in their local copy.

How This Works in Production

Let’s walk through what happens when you update your email address in a system using leader-follower replication with three nodes: one leader and two followers.

Your application sends the update request to the leader. The leader writes “Change user_id=789 email to new@example.com” into its replication log as entry 12,345, saves it to local disk (typically taking under 1 millisecond), and immediately sends this log entry to both followers over the network. Each follower receives the entry, applies it to their local database in order, and sends back an acknowledgment.

Here’s where the crucial trade-off appears: does the leader confirm your update immediately after saving locally (asynchronous replication), or does it wait for at least one follower to acknowledge (synchronous replication)? Asynchronous mode gives you sub-millisecond response times but risks losing recent writes if the leader crashes before followers replicate them. Synchronous mode adds one network round trip (typically 1 to 5 milliseconds within a data center) but guarantees your data exists on multiple machines before you get confirmation.

Most production systems use semi-synchronous replication: they wait for one follower to acknowledge for durability, while keeping additional followers asynchronous for read scaling. MySQL and PostgreSQL commonly run this way, achieving under 10 millisecond write latency while protecting against data loss. The followers can then serve read queries, distributing load across the cluster. If the leader fails, the system promotes one of the up-to-date followers to become the new leader, typically completing this failover in 10 to 30 seconds.

Key Takeaway

Leader-follower replication provides a clean separation of concerns: one node coordinates writes while others replicate for durability and scale reads. Understanding Leader-Follower Replication is foundational for building scalable systems. Learn more in-depth about Leader-Follower Replication on PrepLoop.io, with 3 detailed cards covering advanced patterns, edge cases, and production scenarios.

Learn more in-depth about Leader-Follower Replication on PrepLoop.io

Every time you book an Uber, update a Google Doc with teammates, or post a message in Slack that instantly appears for everyone, consensus algorithms are quietly ensuring all servers agree on the exact order of events. Without these algorithms, distributed systems would be chaos; servers would disagree about which ride request came first, whose edit won, or what messages were sent. Consensus algorithms like Raft and Paxos are the invisible foundation that makes modern distributed applications reliable and consistent.

What Are Consensus Algorithms?

Consensus algorithms solve a deceptively simple problem: getting multiple unreliable computers to agree on a single sequence of decisions, even when some servers crash or networks fail. This agreement is called “quorum consensus,” and it requires maintaining at least 2f+1 replicas to tolerate f failures.

Here’s why this math matters: with 3 servers, you need 2 to agree (tolerating 1 failure). With 5 servers, you need 3 to agree (tolerating 2 failures). The magic is that any two majorities must overlap in at least one server that “remembers” prior decisions, preventing conflicting choices from being committed.

Consensus systems prioritize two critical properties. Safety ensures the system never commits conflicting decisions—this is absolute and never violated. Liveness ensures the system eventually makes progress when a majority is available—this can be temporarily suspended during network partitions.

Both Raft and Multi-Paxos maintain a replicated, ordered log across servers, but they differ in approach. Multi-Paxos evolved as an optimization of basic Paxos, where a stable leader handles proposals after an initial election. However, the original literature leaves many implementation details unspecified, making it notoriously difficult to implement correctly.

Raft explicitly breaks the problem into three clear components: leader election using randomized timeouts, log replication where the leader manages indexed entries, and membership changes. When followers don’t receive heartbeats within 150-300 milliseconds (in wide-area networks), they start a new election. This clarity is why systems like etcd, Consul, and CockroachDB chose Raft over Paxos.

Real-World Implementation

Google Spanner uses Paxos groups with 5 replicas spread across 3+ regions, accepting 50-200 milliseconds of commit latency for exceptional durability. Each Paxos group manages a shard of data, and Spanner runs thousands of these groups to scale horizontally rather than trying to scale a single consensus group.

The performance characteristics are predictable: a 3-node cluster in a single availability zone with NVMe storage achieves 2-6 milliseconds p50 latency. This breaks down to roughly 0.2-2ms for the leader’s fsync (durably writing to disk), 0.5-1ms for network transmission, and 0.2-2ms for follower fsync, plus protocol overhead.

Cross-region deployments face stark trade-offs. Multi-zone configurations within one region add only 1-2ms round-trip time while protecting against zone failures—an excellent balance for most applications. Cross-region setups sacrifice latency (70-100ms coast-to-coast, 150-250ms transoceanic) but provide resilience against entire region outages.

Production systems scale throughput by sharding data across independent consensus groups, each with its own leader. Within a group, batching small entries amortizes fsync costs, but batches must complete within 10-50ms to keep tail latencies acceptable.

Key Takeaways

Consensus algorithms enable distributed systems to maintain consistency despite failures. The choice between 3 and 5 replicas directly impacts both availability (how many failures you tolerate) and performance (how many acknowledgments you wait for).

The fundamental trade-off is geographic: single-region deployments offer low latency but regional vulnerability, while multi-region setups provide disaster resilience at the cost of write latency. Most systems start with multi-zone single-region deployments for the best balance.

Understanding Consensus Algorithms (Raft, Paxos) is foundational for building scalable systems. Learn more in-depth about Consensus Algorithms (Raft, Paxos) on PrepLoop.io, with 3 detailed cards covering advanced patterns, edge cases, and production scenarios.

Learn more in-depth about Consensus Algorithms (Raft, Paxos) on PrepLoop.io

Core Concept: Two Different Worlds of Data Processing

Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) represent two fundamentally different approaches to handling data, each optimized for completely different workload shapes and business requirements.

OLTP systems power the operational heart of applications—the features users interact with directly. When you post a photo on Instagram, update your Slack status, or place an order on Amazon, you’re triggering OLTP transactions. These systems are designed for small, fast operations that read or write just a handful of records. The key characteristics are speed and consistency: OLTP transactions typically complete in single-digit milliseconds and must maintain strict correctness guarantees because they affect real-world state that users can immediately see.

The data modeling in OLTP systems heavily favors normalization. Customer information lives in one table, orders in another, and they’re connected through foreign key relationships. This approach minimizes write amplification—when a customer updates their address, you only change one record instead of updating potentially millions of order records. Every piece of data has a single source of truth, and the system maintains referential integrity through constraints and transactions.

OLAP systems serve an entirely different purpose—they’re built for analysis and decision-making. Instead of handling millions of tiny transactions, OLAP queries might scan billions of rows to calculate revenue trends, identify customer segments, or detect fraud patterns. These queries often take seconds or minutes to complete, but they process vastly more data than any single OLTP transaction ever would.

Where OLTP prioritizes normalization, OLAP embraces denormalization. Data gets flattened into star or snowflake schemas where dimension information is often duplicated directly into fact tables. This redundancy eliminates the need for expensive joins during query time. When analyzing sales by region, you don’t want to join orders to customers to addresses—you want the region information readily available in the sales fact table for immediate aggregation.

The storage architectures reflect these different priorities. OLTP uses row-oriented storage where entire records are stored together, making it fast to retrieve all information about a specific order or user. OLAP uses columnar storage where each attribute is stored separately, allowing queries to read only the columns they need. This dramatically reduces I/O when calculating something like total revenue—you only read the revenue column, ignoring the dozens of other attributes in each record.

The critical insight is workload isolation. Running analytical queries directly against your production OLTP database is a recipe for disaster. A single analyst query scanning millions of rows can exhaust memory, spike CPU usage, and increase transaction latencies from 20 milliseconds to 500 milliseconds, potentially triggering cascading failures that impact user-facing features.

This is why successful architectures maintain a clear separation: OLTP systems own the source of truth and handle user-facing operations, while separate OLAP systems receive data through change streams or batch exports. The OLTP system remains focused on serving users quickly and consistently, while the OLAP system can optimize for different access patterns without interfering with production workloads.

When to Use This Pattern: Choosing the Right Architecture

The OLTP/OLAP split becomes essential when you’re serving both user-facing transactions and analytical workloads that could interfere with each other. If your application only handles operational data with minimal reporting needs, you might run everything on a well-tuned OLTP database with occasional read replicas for lighter analytical queries.

Scale indicators help determine when separation becomes necessary. If your analytical queries regularly scan more than millions of rows, take longer than a few seconds to complete, or run frequently enough to impact transactional performance, it’s time to consider dedicated OLAP infrastructure. When your business users request historical reporting, trend analysis, or complex aggregations across large time ranges, columnar analytical databases will significantly outperform transactional systems.

Data freshness requirements drive architectural decisions. Real-time operational dashboards, fraud detection, and supply-demand matching need streaming CDC pipelines that move data in seconds. Financial reporting, marketing analysis, and strategic planning often work perfectly well with nightly batch exports that reduce system complexity while meeting business needs.

Team and organizational factors matter significantly. Maintaining separate OLTP and OLAP systems requires expertise in different technologies, monitoring approaches, and operational practices. Smaller teams might prefer unified platforms that handle both workloads reasonably well, accepting some performance compromises for operational simplicity.

Cost considerations vary dramatically between approaches. OLTP systems scale with write amplification and provisioned IOPS, while OLAP systems often charge based on data scanned and compute resources consumed. Understanding your query patterns helps optimize costs—heavily aggregated data with predictable access patterns works well with pre-computed materialized views, while ad-hoc exploration benefits from flexible columnar scanning.

The decision often comes down to whether your analytical workloads are predictable enough to isolate through time-based scheduling, resource limits, and read replicas, or whether they’re diverse and intensive enough to warrant separate infrastructure. Most growing applications eventually hit the point where this separation becomes necessary for both performance and operational reasons.

Key Takeaways & Next Steps

Understanding OLTP versus OLAP is fundamental to designing systems that serve both users and business intelligence effectively. Remember that OLTP optimizes for fast, consistent transactions while OLAP optimizes for analytical throughput across large datasets. Workload isolation prevents analytical queries from impacting user-facing performance, typically achieved through CDC pipelines that move data from transactional to analytical systems. The architecture choice depends on your scale, freshness requirements, and team capabilities—start simple but plan for the eventual separation as your system grows.

Ready to dive deeper into database architectures and distributed systems patterns? Explore the complete OLTP vs OLAP learning path on PrepLoop.io, featuring detailed technical cards covering advanced implementation patterns, failure scenarios, and production optimization strategies used by top tech companies.

Learn more in-depth about OLTP vs OLAP on PrepLoop.io