Design systems from first principles, then prove they work.
A study system for platform-engineering system-design interviews — and the prototyping rounds that increasingly come with them. The thinking framework, the building blocks, the numbers you should have memorized, and a playbook for backing every claim with a running service.
The game
Platform interviews are not trivia contests. They test whether you can take a fuzzy need, derive hard constraints from it, decompose a system at the right seams, and reason honestly about what breaks. Prototyping rounds test the same thing, but with running code as the evidence.
System design
Open-ended: "design a multi-tenant PaaS", "design our CI/CD". Scored on constraint derivation, capacity math, component decomposition, failure-mode reasoning, and tradeoff honesty — not on remembering architecture diagrams.
Prototyping / build
Live or take-home: build a small working slice — a rate limiter, a scheduler simulation, a deploy pipeline. Scored on whether it runs, how you structure it, and how you narrate decisions while building.
Deep dive
Pick one domain you claim on your CV — Kubernetes, networking, observability — and get drilled to the config level. You need depth in at least two of the eight blocks below.
Experience & behavioral
Incidents you ran, systems you operated, tradeoffs you owned. Expect "tell me about a time your platform broke" and follow-ups about the mechanism, not the story.
What interviewers actually score
First-principles framework
First principles means: don't reason from analogy ("Netflix uses Cassandra, so we should") — reason from constraints. A design is only defensible if every component can be traced back to a constraint, a number, or a failure mode. This sequence is the whole method; run it in order and you will not get lost in any system-design round.
Pin the constraints
Users (active, concurrent, growth), traffic (RPS per operation, read:write ratio, payload sizes), data (volume, retention, consistency needs), latency targets, availability targets (and who defined them), cost ceiling, team size, and the time-to-ship. Write them down before anything else.
Derive requirements, don't list them
Each requirement must trace to a constraint: "we need 4 nines" ← the business SLA; "reads must be <100ms p95" ← the UX budget; "tenants must not see each other" ← the security/compliance requirement. If a requirement traces to nothing, cut it.
Capacity math first
Before any component: QPS per operation, storage per unit × growth × retention, bandwidth, instance counts via Little's law (L = λW), replication overhead. This tells you the real shape of the system — whether you need 3 instances or 30, one database or sharding.
Decompose at the seams
Split the system at boundaries of ownership, scale rate, failure domain, and team. Each component gets: what it owns, what it guarantees, what it depends on, what happens when it fails. If two components always scale together and fail together, they're one component.
Define interfaces before implementations
API shapes, contracts, protocols, and data models between components — before choosing technologies. "This is a queue with at-least-once delivery and idempotent consumers" is a design; "we'll use Kafka" is an implementation detail you can now justify.
Walk the critical path
Trace one request end-to-end: every hop, every write, every wait, and its latency contribution. Then do the same for the failure path. Sum the latencies against your budget; the critical path is where you spend your design budget.
Enumerate failure modes, pick mitigations
For each component: crash, slowdown, saturation, partition, operator error, noisy neighbor. For each: blast radius, detection signal, mitigation. Prioritize by probability × impact — mitigate the top two or three, state the rest as accepted risk.
Deep-dive where the risk is
Pick the 2–3 components where correctness, cost, or scale actually hinges — and design those to the config level: sharding scheme, quorum sizes, retry policy, eviction policy. Everything else stays at one line.
Prototype the risky claim
In a build round, implement the one claim that carries the design: the scheduler loop, the rate limiter, the fan-out. A 100-line Go service that runs beats a 500-line sketch that doesn't compile. Section 04 is the playbook.
"Deploy 500 services, commit → production in 5 minutes"
The framework applied in ~30 seconds — this is the shape of a strong opening move.
- Constraints: 500 services, say 20 commits/min peak, 5-min deploy budget, deploys must be reversible in <5 min, 50 engineers.
- Capacity: 20 commits/min × 500 services is trivial queue load; the bottleneck is build (1–3 min) and rollout, not the pipeline machinery. A single queue + pool of 20–50 build workers covers it. Math: 20 builds/min × 2.5 min/build ≈ 50 concurrent builds → 50 workers, cheap.
- Seams: trigger → build+test → package+sign → promote artifact → rollout (canary) → verify → rollback. Promotion and rollout are separate components — artifacts are immutable, deploys are reversible.
- Critical path: commit → trigger (<1s) → build (~2 min) → promote (~10s) → canary 1% (~1 min) → full rollout (~1 min). Total ~4 min. The risk is the canary gate, not the queue.
- Failure modes: build flake (retry with cap), bad commit (canary metrics auto-halt + rollback to last good artifact), queue saturation (priority per service, cap in-flight per repo), registry outage (promotion fails closed — deploys pause, don't degrade).
- Deep dives: canary evaluation (what metric, what window), rollback mechanics (artifact pin, not re-build), artifact signing.
Building blocks
Eight domains you will be asked about. For each: the essence, the first principles underneath it, the numbers to know, and the questions to expect. Don't memorize — derive.
Compute & scheduling
Turning a fleet of machines into a place where workloads run predictably. At the core: a scheduler that maps "what a pod needs" onto "what a node has", plus isolation so neighbors can't hurt each other.
- Scheduling is constraint satisfaction + a packing heuristic: filter (resources, taints, affinities) then score (spread vs pack).
- requests drive scheduling; limits drive throttling. Overcommit CPU, never memory — OOM kill is worse than throttle.
- Isolation is cgroups + namespaces; a "noisy neighbor" is just contention on an oversubscribed resource.
- Every scheduler decision has a revocation cost (evict, drain, reschedule) — prefer soft constraints where eviction is expensive.
Networking
Getting packets from A to B with the right guarantees. In platforms: L4 vs L7 load balancing, service discovery, ingress, east-west traffic between services, and identity via mTLS.
- L4 routes connections (cheap, dumb, per-conn); L7 routes requests (TLS termination, retries, content routing — expensive, smart). Use each where its cost is earned.
- DNS is eventual: TTL means failover lags 30–300s. Anything that needs instant failover needs a proxy or L4 LB in front.
- Service discovery turns names into healthy endpoints; health must be checked from the caller's perspective.
- A service mesh buys uniform mTLS/policy across a large polyglot fleet — and costs an extra network hop and a control plane. Under ~20 services, plain mTLS + ingress usually wins.
Storage
State, in its three shapes: block (attach to one node), file (shared over a protocol), object (S3: HTTP, cheap, massively durable). Plus the hard part: replication and consistency for stateful systems.
- Object storage wins whenever you don't need low-latency random writes; 3× replicate or erasure-code for durability, not for speed.
- Quorum math: N replicas, writes need W > N/2, and R + W > N so every read sees a fresh write. Majority quorum means the cluster survives N/2 − 1 failures.
- etcd/consul are the platform's own database: raft, odd cluster sizes (3/5/7), and the watch API is how controllers work.
- Stateful workloads in k8s need local/network disk, pod↔volume affinity, and backup/restore as a first-class path — not an afterthought.
Observability
Signals to detect, triage, and budget for failure. Metrics (aggregates), logs (events), traces (one request across services), and SLOs tying it to business reality.
- RED for services (Rate, Errors, Duration); USE for resources (Utilization, Saturation, Errors). Four golden signals: latency, traffic, errors, saturation.
- SLO chain: pick SLI → set SLO → error budget = 1 − SLO. Budgets fund deploys and experiments; burn-rate alerts fire on fast budget spend.
- Alert on symptoms (p95, error rate), not causes ("CPU high" is a cause).
- Traces need sampling at volume (0.1–1% tail-safe sampling); watch cardinality — labels like user-id explode metric series.
Delivery
Turning code into production safely and reversibly: pipelines, GitOps, progressive delivery, rollback. The platform's most visible product.
- CI = build+test on change; CD = promote artifacts, never rebuild. Immutable, signed artifacts move through environments.
- GitOps: an in-cluster agent reconciles desired state from git (pull) — auditability and no CI-held cluster creds, at the cost of an agent per cluster.
- Canary: shift 1→5→25→50→100% on metrics gates — cheap capacity, catches bad code on real traffic. Blue-green: instant switch, but 2× capacity and state-migration pain.
- Deploys must be idempotent and reversible; rollback = point at last good artifact, <5 min.
Multi-tenancy
Running many customers on one platform without letting them hurt each other — the defining platform problem. Isolation is a spectrum, and it costs money.
- Hard isolation (VMs, dedicated clusters) costs capacity; soft isolation (namespaces, quotas, rate limits) costs engineering. Match isolation to risk, not to fear.
- Shared control plane, isolated data plane is the sweet spot: API/console shared; each tenant's workloads and data fenced by namespace + network policy + quota.
- Noisy neighbors attack whatever's oversubscribed: CPU steal, IO, memory pressure, a shared database. Quota everything — resources, objects, and requests per tenant.
- Blast radius thinking: what does one compromised or buggy tenant take down? That answer defines your isolation level.
Autoscaling
Matching capacity to load — three loops: scale pods (HPA), right-size requests (VPA), scale nodes (cluster autoscaler). Scaling is slow, so you buy headroom or you buy p95s.
- HPA: desired = ceil(current × target/targetMetric), with a stabilization window so jitter doesn't flap replicas.
- VPA changes requests (needs pod restarts); CAS adds nodes only when pods are unschedulable — and scales down carefully (eviction + PDB + pod-disruption safety).
- Scale-up takes minutes (node boot + image pull); your headroom is your tolerance for that latency. Overprovision deliberately at the base.
- Scale the control plane too: API server, scheduler, and etcd all have ceilings when cluster size or churn grows.
Security
Least privilege, identity, secrets, and a supply chain you can audit. In platforms: assume breach, minimize blast radius, make secrets boring.
- Identity first: RBAC (roles → bindings, least privilege), mTLS between services, short-lived workload identity over static creds.
- Secrets: KMS envelope encryption, external stores + CSI/sidecar injection, rotation as a routine. Never in env vars, images, or CI logs.
- Supply chain: signed images (cosign), SBOMs, admission policy (OPA/Kyverno) that blocks unsigned or non-compliant workloads, default-deny network policies.
- The platform's own control plane is the crown jewel — its compromise is every tenant's compromise.
Prototyping playbook
Build rounds score one thing: did you prove the risky claim with running code? Every snippet on this page was compiled and executed as part of building this site — copy them freely, they run.
The rule: one claim, thinnest slice
Pick the single claim your design hinges on — "the scheduler converges under churn", "the rate limiter holds the ceiling" — and build the thinnest vertical slice that demonstrates it. Timebox: 30 min build, 15 min walkthrough. Narrate while you type: state the claim, show the code, run it, read the evidence.
Say out loud what the prototype must prove. If it doesn't prove that, it's scope creep.
One service, one path, real endpoints, real metrics. Stdlib only where possible — zero-dependency code impresses more than a scaffolded framework.
curl the happy path, curl the failure path, load it, show the metric move. Evidence is part of the answer.
Show what breaks: kill it mid-load, exhaust the limiter. Interviewers remember the failure demo.
the skeleton — go, stdlib only
A service with /healthz, /readyz, a working endpoint, and in-memory p95 — the base for any prototype. Verified: compiles, runs, serves, and its p95 moves under load.
package main
import (
"fmt"
"log"
"math/rand/v2"
"net/http"
"sort"
"sync"
"sync/atomic"
"time"
)
type window struct {
mu sync.Mutex
seen []time.Duration
}
func (w *window) add(d time.Duration) {
w.mu.Lock()
defer w.mu.Unlock()
w.seen = append(w.seen, d)
if len(w.seen) > 4096 { // bounded ring
w.seen = w.seen[len(w.seen)-4096:]
}
}
func (w *window) p95() time.Duration {
w.mu.Lock()
defer w.mu.Unlock()
if len(w.seen) == 0 {
return 0
}
s := make([]time.Duration, len(w.seen))
copy(s, w.seen)
sort.Slice(s, func(i, j int) bool { return s[i] < s[j] })
return s[int(float64(len(s))*0.95)]
}
func main() {
win := &window{}
ready := &atomic.Bool{}
ready.Store(true)
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
http.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
if ready.Load() {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusServiceUnavailable)
})
http.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
time.Sleep(time.Duration(5+rand.IntN(40)) * time.Millisecond) // simulated work
win.add(time.Since(start))
fmt.Fprintf(w, "done in %v\n", time.Since(start))
})
http.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "window_samples %d\nwindow_p95_seconds %.4f\n", len(win.seen), win.p95().Seconds())
})
log.Println("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}ship it — minimal, hardened
# syntax=docker/dockerfile:1
FROM golang:1.23-alpine AS build
WORKDIR /src
COPY go.mod ./
COPY main.go ./
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/server .
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/server"]apiVersion: apps/v1
kind: Deployment
metadata:
name: proto
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels: { app: proto }
template:
metadata:
labels: { app: proto }
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector: { matchLabels: { app: proto } }
containers:
- name: proto
image: registry.example.com/proto:1.0.0
ports: [ { containerPort: 8080 } ]
resources:
requests: { cpu: 100m, memory: 64Mi }
limits: { cpu: 500m, memory: 128Mi }
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
initialDelaySeconds: 2
periodSeconds: 3
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
initialDelaySeconds: 5
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: proto
spec:
selector: { app: proto }
ports: [ { port: 80, targetPort: 8080 } ]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: proto
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: proto }
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }the evidence script
- Start it: go run main.go → it logs and binds :8080.
- Happy path: curl -s localhost:8080/work → done in 23ms-style reply. curl localhost:8080/readyz → 200.
- Load it: hey -z 30s -c 100 http://localhost:8080/work → watch /metrics: samples climb, p95 settles near the true workload p95.
- Failure demo: kubectl delete pod mid-load → readiness gates traffic, new pod takes over. Narrate the gap.
- Close the loop: restate the claim, point at the number that proved it.
what kills prototype rounds
Over-building
Frameworks, config layers, generated clients — 20 minutes of scaffolding before a single curl. Slice first; nothing earns its place unless it serves the claim.
Silent failure demos
Running code that breaks while you keep talking. If something errors during the demo, stop, read the error, and debug out loud — that's the whole interview, and it scores well.
No numbers
A prototype without a measured result is a screensaver. Always end on: request rate, p95, error count — before and after.
Uncommitted state
If it's a take-home: git history that shows the thinking, a README that states the claim and how to run it, and a short Loom-style walkthrough. Reviewers read these first.
Practice problems
Five realistic platform problems. For each: how the interviewer frames it, the constraints to pin, an approach, example capacity math, failure modes, and what you'd prototype. Try each cold in 45 minutes before opening the approach.
pin first
Tenant count and size distribution (10 big vs 10,000 small changes everything), isolation requirement (compliance? same-VM acceptable?), what tenants can deploy (containers only? custom images?), network model, and who's billed for what.
approach
- Control plane (shared): API + admission + scheduler. Per-tenant auth (OIDC), RBAC scoped to tenant namespaces.
- Data plane (isolated): per-tenant namespace with ResourceQuota + LimitRange, default-deny NetworkPolicy, per-tenant service accounts. If compliance demands, separate node pools or dedicated clusters for the big tenants.
- Image policy: allowlist registries, signature verification at admission; no privileged pods.
- Metering: usage collector (requests, storage, egress) → billing; quota enforcement by admission, not by after-the-fact billing.
- Ingress: per-tenant hostname → ingress controller with per-tenant rate limits (token bucket in front of tenant routes).
capacity math
failure modes
- Noisy neighbor saturates a shared node (mitigate: quotas, spread constraints, pool isolation for large tenants)
- Admission outage blocks all deploys (fail-open vs fail-closed — decide per action, log everything)
- Control plane saturation from tenant churn (rate-limit watch/list calls)
- Tenant compromise → blast radius limited by namespace + network policy + node pool
prototype in 45 min
A tiny admission webhook that enforces per-tenant resource quota, plus a demo: tenant A's deploy is rejected when over quota while tenant B deploys fine. That proves the isolation claim.
pin first
How teams onboard today (pain map), what "golden path" means here (one blessed way, escape hatches for exceptions), service count and deploy rate, and who supports the platform when it breaks.
approach
- Catalog: services as code (name, owner, repo, runbook) — the source of truth everything else reads from.
- Scaffolding: templates that generate repo + CI + k8s manifests + SLO defaults; template engine keeps config in one place, teams own their instance.
- CI: one pipeline template per language/framework; artifacts pushed to a registry with signing.
- CD: GitOps — per-environment state repo; agent reconciles. Promotion = PR between environment branches; approval = merge.
- Environments: dev/ephemeral per PR (spawn on open, destroy on merge), staging, prod. Ephemeral envs are the single biggest DX win.
- Observability default-on: every scaffold ships with metrics + dashboards + alert routing from day one — you can't opt out of being visible.
capacity math
failure modes
- Platform team becomes a bottleneck for exceptions (mitigate: escape hatches + platform-as-library, not platform-as-ticket-queue)
- Ephemeral env leak eats the cluster (TTL reaper + hard quotas)
- GitOps drift when teams kubectl-apply around the reconciler (make the reconciler authoritative, alert on drift)
- Template updates break existing services (versioned templates, pin per service)
prototype in 45 min
Scaffold → commit → CI builds → GitOps reconciler deploys → service reachable. A 100-line controller that watches a git repo and applies manifests proves the whole pull model.
pin first
Repo count and commit rate, build type mix (Go/Java/Node — build times differ 10×), p95 build time target, fan-in (monorepo vs polyrepo), and what "safe" means for untrusted PRs from forks.
approach
- Trigger: webhooks → queue, dedupe per ref (latest commit wins for PRs), priority per repo/team.
- Workers: pool of ephemeral VMs/containers; warm pool for the hot languages; per-build isolated workspace. Jobs declare resource class (small/medium/large) and the scheduler bins them.
- Caching: content-addressed layer cache (registry mirror) + dependency cache keyed by lockfile hash. Cache is the single biggest speed lever.
- Security: fork PRs run in sandboxed runners with no secrets; secrets only for trusted branches; everything else blocked at the controller.
- Outputs: signed artifacts to the registry; results + logs to the API; status back to git.
capacity math
failure modes
- Cache poisoning / eviction storms (content-addressed + immutable layers)
- Queue starvation by one big repo (fair-share scheduling per repo, not FIFO)
- Worker compromise escapes the sandbox (untrusted = fresh VM, no cache sharing)
- Webhook floods during incident redeploys (rate-limit + priority classes)
prototype in 45 min
A tiny dispatcher: webhook → in-memory queue → N worker goroutines that run `go test` in temp dirs, report pass/fail. Load it with 100 fake commits and show queue depth + fairness stats.
pin first
Read volume (every request evaluates flags — this is the hottest path in the company), write volume (rare but urgent), propagation latency target for a kill switch, and who can flip what (audit + approvals).
approach
- Writes: admin API → durable store (postgres or etcd) with versioned history, audit log, and approval for sensitive flags. Writes are rare — optimize for correctness, not throughput.
- Fan-out: each service keeps the full flag snapshot in memory, refreshed by poll (30s) or push (SSE/long-poll/websocket) from a distribution hub. Local eval = zero network on the request path.
- Eval: deterministic, in-process: flag + context (user, tenant, % rollout) → variant. Uniform hash for % rollouts so a given user stays put during the ramp.
- Kill switch: emergency path that bypasses approval and pushes within seconds — separate from normal flag flow.
capacity math
failure modes
- Hub outage → services keep last-good snapshot (stale-but-functional is the correct degradation)
- Flag flip takes 30s+ to propagate — kill switch needs its own fast path
- % rollout hash collisions or unstable hashing (use stable hash of a fixed id)
- Flag explosion without owners/deadlines (flag lifecycle: every flag has an expiry and an owner)
prototype in 45 min
An HTTP config server + a client lib that caches and polls; demo: flip a flag in the server, watch it appear in the client within the poll interval, then cut the server and show the client serving stale config happily.
pin first
Fleet size, workload diversity (long-running vs batch vs spot-tolerant), diurnal/seasonal patterns, lead time to add hardware, and the cost floor (committed-use discounts vs on-demand).
approach
- Observe: per-workload usage history (CPU/mem/IO) at 5-min granularity; forecast with simple models (seasonal + trend), not ML theater.
- Place: bin-packing with constraints (affinity, fault domains, cost classes). Place long-running on reserved/committed, batch on spot, latency-critical on dedicated.
- Scale: three loops (HPA for pods, VPA for right-sizing requests, autoscaler for nodes) with a stabilization layer so they don't fight.
- Account: cost per workload, per team, per environment → showback/chargeback. FinOps is a first-class output, not a monthly surprise.
capacity math
failure modes
- Scaling loops oscillate (stabilization windows + hysteresis on every loop)
- Forecast undershoots a launch spike (caps on per-workload growth + manual override path)
- Spot reclaims take down batch (graceful termination handling, checkpointing)
- Fragmentation wastes 20–30% (smarter bin-packing, pod right-sizing via VPA)
prototype in 45 min
A bin-packing simulator: N workloads with resource shapes, greedy + best-fit placement, report utilization + fragmentation. Add a "kill a node" event and show rebalance. Visually convincing and pure code.
The 45-minute script
A timeboxed game plan for the system-design round. The clock is the constraint that fails most candidates — this allocates it deliberately. (Prototyping-round variant below.)
Restate + interrogate. Repeat the problem back with your own words, then pin constraints: scale, traffic shape, consistency, availability target, who operates it, cost. Ask the four questions: users? traffic? consistency? who runs it?
Numbers + shape. Capacity math out loud (Little's law, storage, bandwidth). Sketch the high-level components at the seams — one line each. State your non-goals. Get the interviewer to nod before going deeper.
Two deep dives. Walk the critical path with latency budget per hop, then the failure modes of the two riskiest components, each with detection + mitigation. Volunteer "what breaks first" before being asked.
Tradeoffs + scaling story. What changes at 10× load: which component gives first, what you'd swap. State the tradeoffs you accepted and why — consistency vs availability, shard now vs later, isolation level.
Close. SLOs you'd set, rollout plan (stages, canary, rollback), 2–3 open questions. Offer a summary in three sentences. A clean close is remembered.
45 minutes to build, 15 to defend
- 0–3 min: restate the problem, state the one claim you'll prove, sketch the slice (endpoints, data, demo script).
- 3–8 min: skeleton running — server up, /healthz answering. Commit.
- 8–30 min: the claim, in small commits: happy path → metric → failure handling. Talk through each commit.
- 30–40 min: load it, read the numbers, tune once. If something breaks, debug out loud — it's content.
- 40–45 min: quick README + walkthrough: claim → code → evidence → what you'd do with a week.
Numbers cheat sheet
Memorize these. They are the vocabulary of every capacity discussion, and interviewers notice immediately when you reach for them fluently. All values are engineering heuristics — order-of-magnitude truth, not specs.
Latency
| operation | time |
|---|---|
| L1 cache reference | ~1 ns |
| Main memory read | ~100 ns |
| Context switch (userspace) | 1–10 µs |
| NVMe SSD random read | 10–100 µs |
| Spinning disk seek | 1–10 ms |
| Same-DC round trip | ~0.5 ms |
| Cross-continent round trip | 100–150 ms |
| 1 Gbps network | ≈100 MB/s |
| 10 Gbps network | ≈1 GB/s |
| k8s pod start (warm image) | 2–10 s |
| Node scale-up | 1–5 min |
Availability
| nines | downtime / year |
|---|---|
| 99% | 3.65 days |
| 99.9% | 8.77 hours |
| 99.95% | 4.38 hours |
| 99.99% | 52.6 min |
| 99.999% | 5.26 min |
redundant: A = 1 − (1−A₁)²
Two 99% components in series = 98.01%. Two 99% in parallel = 99.99%. Every extra nine costs real architecture — ask who's paying.
Capacity
concurrency = RPS × avg latency
| rule | value |
|---|---|
| stateless service per instance | ~0.5–2k RPS |
| well-sized API server | ~1–3k QPS |
| postgres (tuned) per node | ~2–5k TPS |
| etcd cluster size | 3 / 5 / 7 (odd) |
| standard replication | 3× storage |
| k8s node bin-packing loss | 20–30% |
Quorum & consensus
reads need R + W > N
With N=3: W=2, R=2. The cluster tolerates 1 node down. With N=5: tolerates 2. Majority quorum is why raft clusters use odd sizes — an even split wastes a node without adding failure tolerance.
99.9% monthly = 43.8 min of budget/month. Burn rate 10× (spending 43.8 min in 4.4 min) = page on-call.
Capacity calculator
Downtime budget
What a given SLO allows you to spend per year, month, and week — the error budget you deploy against.