Putting It Together — Designing a Real System & Trade-offs

This is the capstone section. Until now you have learned five separate topics: databases (where data lives), caching (keeping copies of data close by for speed), networking (how machines talk), concurrency (doing many things at once safely), and distribution (spreading work across many machines). In real life you do not use these one at a time. You pull them all together, like levers on a control panel, while designing one system.

The good news: you do not need to memorize famous architectures. You need a repeatable thinking process. Learn the process and you can reason about any system — even one you have never seen before.

The repeatable design process

Follow these seven steps in order, every time:

1. Clarify functional requirements — what must the system do? (Example: shorten a long web address into a short one, and redirect the short one back.)
2. Clarify non-functional requirements — how well must it do it? This means scale (how many users), latency (how fast), availability (how often it is up), consistency (how fresh/correct the data must be), and durability (will we lose data).
3. Back-of-envelope estimate — rough math for QPS, storage, and bandwidth. (QPS = "queries per second," the number of requests hitting the system each second.) This tells you whether one computer is enough or you need many.
4. Define the API and data model — the contract. What does the client send, what comes back, and how is data stored.
5. Sketch the high-level design — boxes and arrows: client → load balancer → app servers → cache → database, plus background workers.
6. Find the bottleneck and scale it — the bottleneck is the slowest, most overloaded part. Usually it is the database or the hot reads.
7. Harden for reliability — redundancy, graceful degradation, idempotency, monitoring.

Back-of-envelope estimation: the cheat sheet

"Back-of-envelope" means quick rough math you could do on the back of an envelope. Memorize these handful of facts and you can estimate almost anything.

• Seconds in a day ≈ 86,400 ≈ 10⁵ (one hundred thousand). So 1 million per day ≈ ~12 per second on average. Flip side: 1 request/second ≈ ~2.5 million/month.
• Peak ≈ 2–10× the average. Traffic is bursty (everyone shops at lunchtime). Always state your peak multiplier.
• Bandwidth = average payload size in bytes × QPS. (Multiply by 8 if you want bits/second to quote Mbps.)
• Storage = records per day × bytes per record × how many days you keep it × replication factor (how many copies you store).
• Power-of-two sizes: 2¹⁰ ≈ 1 KB, 2²⁰ ≈ 1 MB, 2³⁰ ≈ 1 GB, 2⁴⁰ ≈ 1 TB. One ASCII character = 1 byte; a UUID = 16 bytes.
• Rough single-node ceilings (sanity bounds, not gospel): a relational database does ~10,000 writes/second and tens of thousands of reads/second per machine; Redis (an in-memory cache) does 100,000+ operations/second.

Worked example A: a URL shortener (read-heavy)

A URL shortener takes a long web address and gives back a short code (like the ones you see in text messages). Click the short code and it redirects you to the original.

Requirements: create a short code from a long URL; redirect a short code to its long URL. Assume ~100 million new URLs per day, and links must work for years.

Estimation:

• Writes: 100,000,000 ÷ 86,400 ≈ ~1,160 writes/second average. With a peak multiplier of ~10×, call it ~10,000/second at peak.
• Reads: a URL shortener has a read:write ratio around 100:1 (each link is created once but clicked many times). So ~116,000 reads/second average — much higher at peak.
• Storage: ~500 bytes per record (long URL + metadata) × 100M/day × 365 days × 5 years ≈ ~90 TB over five years.
• Short code length: 7 characters using Base62 (the 62 URL-safe characters: a–z, A–Z, 0–9) gives 62⁷ ≈ 3.5 trillion possible codes — decades of headroom. We use Base62, not Base64, because Base64 includes + and /, which have special meaning inside URLs.

Design insight: with 100 reads for every write, the cache is the system. Generate a unique ID (from a global counter, or a Snowflake-style 64-bit ID), Base62-encode it as the short code, and store code → long_url. The redirect is just a primary-key lookup, so cache it hard: a CDN (a network of edge servers near users) plus Redis in front of the database.

Worked example B: e-commerce checkout (write-heavy, money on the line)

Browsing a catalog is read-heavy (cache it). Checkout is the hard part because real money, inventory, and outside services are involved. Every topic from earlier shows up here with teeth.

• Concurrency — the "last item in stock" race: two buyers click "Buy" for the final unit at the same moment. Without protection, both succeed and you oversell. The fix is a database transaction with row locking (SELECT ... FOR UPDATE, which "reserves" the row so the second buyer waits) or an atomic decrement (a single uninterruptible "subtract one"). This is the classic lost update problem made concrete.
• Consistency vs. availability, per piece: inventory and payment need strong consistency — you must never oversell or double-charge. But the product's review count can be eventually consistent — it is fine if it is a few seconds stale. Different parts of one system sit at different points on the spectrum. That is the key insight.
• External calls + retries: the payment gateway might time out after it already charged the card. The client retries and — without protection — charges twice. This is exactly why idempotency keys exist (below).
• Async work: the confirmation email, invoice PDF, and warehouse notification go on a queue (a waiting line of jobs processed by background workers), not on the request path. This keeps checkout fast and survives a downstream outage.

        ┌── CDN (static files, images)
Client ─┤
        └─ LB ─ App servers ─ Cache (Redis) ─ DB (primary + replicas)
                     │
                     └─ Queue ─ Workers (email, PDF, inventory sync)

  LB = load balancer (spreads requests across app servers)

Latency numbers every programmer should know

Latency means "how long one thing takes." These rounded numbers (the classic Jeff Dean / Peter Norvig table) drive almost every design decision:

The mental model: memory is roughly 100× faster than SSD, which is roughly 100,000× faster than a cross-continent network round trip. This is why we cache in memory and why we avoid chatty cross-region calls.

The five universal trade-offs

This is the heart of the section. Almost every design decision is one of these tugs-of-war.

Reliability: making it survive failure

Redundancy — no single point of failure

A "single point of failure" is one part whose death takes down everything. Avoid it: run multiple app servers behind a load balancer, keep database replicas with failover, and spread across availability zones or regions. But redundancy only helps if failover is automatic and tested. An untested standby gives false confidence — and remember the load balancer, the cache, and the database can each themselves be a single point of failure.

Graceful degradation

Under heavy stress, drop non-essential features instead of falling over completely. Serve a slightly stale cached page, hide recommendations, disable the review widget — but keep checkout working. A worse-but-working experience beats a blank error page.

Idempotency keys and safe retries

An operation is idempotent if doing it twice has the same effect as doing it once. To make "charge the card" idempotent, the client generates an idempotency key (a unique UUIDv4) per logical request and sends it along. The server stores key → result. If a retry arrives with the same key, the server returns the stored result instead of charging again. (Stripe keeps these keys for ~24 hours.)

Retries should use exponential backoff with jitter. The formula:

delay = min(cap, base × 2^attempt) + random_jitter

Observability: the three pillars

Observability means being able to understand what your system is doing from the outside. It rests on three pillars:

• Metrics = numeric time-series (QPS, error rate, p99 latency, CPU). Cheap; perfect for dashboards and alerts. They tell you what is wrong.
• Logs = timestamped records of events, with rich detail. They tell you why it is wrong.
• Traces = the journey of one request across all the services it touches, with timing at each hop. They tell you where it is slow or failing.

They work together: metrics alert, traces localize, logs explain. OpenTelemetry (OTel) is the vendor-neutral standard for emitting all three.

SLI, SLO, SLA, and error budgets

• SLI (Service Level Indicator) = the measured number, e.g. "% of requests served under 200 ms."
• SLO (Service Level Objective) = your internal target for that SLI, e.g. 99.9%.
• SLA (Service Level Agreement) = a contractual promise to customers, with penalties (refunds) if you miss it. It is usually set looser than the SLO, so you have a safety margin.
• Error budget = 100% − SLO = how much failure you are allowed. A 99.9% monthly SLO gives ~43 minutes of budget. Spend it on shipping faster and taking risks; when it is used up, freeze risky changes.

Each extra nine roughly costs 10× more to achieve. Pick the level your business actually needs — most products do not need five nines.

How to reason about any system you meet

This is the takeaway skill. When you face a system you have never seen, ask these questions in order:

1. What does it do?
2. What is the scale (QPS, storage)?
3. What is the read:write shape?
4. Where is the bottleneck?
5. What does it cache, and what is the invalidation (staleness) story?
6. What is the consistency requirement for each piece?
7. What happens when X fails?
8. How would I know it failed (metrics/alerts)?

If you can answer those eight questions, you can analyze any system. There is rarely one "right" design — only trade-offs made explicit and matched to requirements.