Distributed systems interviews aren't about systems

I watched a senior engineer fail a system design interview last year. He knew everything. Consistent hashing, vector clocks, the exact tradeoffs between Cassandra and DynamoDB. His whiteboard looked like a textbook. He didn't get the offer.

Afterward, I pulled the interviewer aside. What happened? She said: "He never once asked me what the product actually needs to do."

That's the whole thing right there.

The question behind the question

When someone asks you to design a rate limiter at scale, they're not running a distributed systems quiz. They're watching how you think when you don't have enough information. Which is every single day of your actual job.

In 2021, our team at a previous company (around 800 engineers, processing about 2M transactions/day) spent three weeks building the wrong caching layer because nobody stopped to ask what "fast" meant to the product team. We assumed sub-50ms. They needed sub-500ms but with 99.99% consistency guarantees. Those are completely different architectures. That mistake cost us roughly six weeks of rework and pushed a key merchant launch by a quarter.

The distributed systems concepts you study are real. They matter. But they're vocabulary, not the answer.

What interviewers are actually watching

Here's what changes when I sit on the other side of the table as a manager who's interviewed maybe 200 engineers over three years:

I'm watching for how you handle ambiguity. Do you freeze and start drawing boxes? Or do you say "before I get into the architecture, can I understand the read/write ratio you're expecting?"

I'm watching when you introduce complexity. A lot of candidates hear "design a distributed cache" and immediately start talking about consistent hashing and replication factors. Strong engineers say "do we even need distribution here, or can a single Redis instance with a read replica get us 90% of the way?"

That instinct — to reach for the simplest thing that works — is rare. It's also what separates good engineers from great ones in actual production environments.

The concepts that actually come up (and why)

CAP theorem comes up not because you need to recite it but because it forces a conversation: what breaks in your system and what doesn't? When I ask about it, I'm hoping you'll say "it depends on whether our users can tolerate seeing stale data for 30 seconds" — not that you'll define partition tolerance.

Idempotency comes up constantly. More than any other concept. Because almost every real bug I've seen in distributed systems comes down to "we processed that event twice." If you can talk concretely about how you'd design an order processing system to be idempotent, using something like a deduplication_key in Postgres or a conditional write in DynamoDB, you're showing production experience, not textbook knowledge.

"Tell me about a time something failed and you had to trace it across services." — I ask this in almost every interview now. The answer tells me more about someone than an hour of system design.

What to do differently starting tomorrow

When you practice system design, stop when you've drawn the first box and ask yourself: what would break first, who would notice, and how would we know? That's the interview. That's also the job.

When I was an IC, I thought the goal was to show how much I knew. As a manager, I now see that the goal is to show how you'd behave when things go sideways at 2am and Datadog is showing five different anomalies at once. Design your answer around that engineer. That's the one who gets hired.