Learn / Prove it — practice ladder

Load-test it and fix the bottleneck

Lesson 37 of 37 · 9 min read ·

The brief

Take something you built, put load on it until it breaks, find the actual bottleneck, fix it, and measure again. The loop — measure, find, fix, re-measure — is the whole skill. Doing it once end to end teaches more about performance than any amount of reading.

Before you start: define the goal

"Make it fast" is not a target. Write down a specific one:

200 concurrent users browsing the catalogue, p95 under 300 ms, error rate under 0.1%.

Without a number you cannot tell success from noise, and you will optimise something that did not matter.

Set it up honestly

Tools: k6 (JavaScript, excellent output), Vegeta (simple, CLI), Locust (Python), or autocannon for a quick single-endpoint check. Any of them is fine.

The environment is where people fool themselves:

  • Test against something production-like. Results from your laptop, against SQLite, with 100 rows of data, are worthless.
  • Seed realistic data volume. A query against 500 rows tells you nothing about the same query against 5 million. Most bottlenecks are invisible at small data sizes, which is the entire reason they reach production.
  • Generate the load from a different machine. Otherwise the load generator competes with the app for CPU and you are measuring your own tool.
  • Model a real journey, not one endpoint on repeat. Users log in, browse, search, then write.
  • Ramp up gradually. A step increase tells you the breaking point; a ramp tells you the shape of the degradation.

Run three tests

  1. Baseline. Realistic load, sustained. Record p50, p95, p99, throughput and error rate.
  2. Stress. Ramp until it degrades. Find the exact point where p99 leaves the floor and errors appear. That number is your capacity — knowing it is more valuable than any single optimisation.

The knee: throughput stops rising and latency takes off. That number is your capacity.The knee: throughput stops rising and latency takes off. That number is your capacity. 3. Soak. Moderate load for an hour or more. This is what catches memory leaks, connection leaks and file-descriptor exhaustion — none of which appear in a five-minute run.

Find the bottleneck — do not guess

Guessing is the default behaviour and it is almost always wrong. People rewrite a JSON serialiser while the database is doing a sequential scan.

Work down the stack with evidence:

  1. Is it saturation? CPU, memory, connection pool, thread pool, disk I/O. Whichever is at 100% is your constraint. A pool at capacity while CPU sits at 20% means requests are queueing for a connection, and adding CPU will do nothing.
  2. Where does the time go? A trace of one slow request under load answers this immediately: 15 ms in the handler, 780 ms waiting on the database, 20 ms serialising. See observability.
  3. Which query? Enable slow query logging, take the worst one, run EXPLAIN ANALYZE. See query plans.
  4. How many queries? Count queries per request. If one request makes 143, you have an N+1 and no index will save you.

The usual suspects, in order of frequency

  1. A missing index — a sequential scan that was fine on dev data.
  2. N+1 queries from an ORM.
  3. Connection pool too small, so requests queue invisibly.
  4. No caching on an expensive, rarely-changing read.
  5. Synchronous work that should be queued.
  6. No timeouts, so one slow dependency backs everything up.
  7. Serialising far more data than the client uses.

Note how far down the list "the code is slow" appears. Application CPU is rarely the first bottleneck in a web service.

Fix one thing, then measure again

Change exactly one thing and re-run the same test. Two changes at once means you do not know which worked — and sometimes one helped while the other hurt.

Expect the bottleneck to move. Fix the index and the database drops to 5%; now the connection pool is the limit. That is success, not failure. Keep going until you hit the target or the next fix is not worth its cost.

Write down the numbers before and after. "p95 went from 2.1 s to 180 ms by adding a composite index on (user_id, created_at)" is a sentence worth having.

Prove you know it

Run the loop once, completely: baseline, find, fix, re-measure. Then answer the question a load test exists to answer — how many concurrent users can this handle before users notice? If you can state that number and name what breaks first, you have the skill.

Go deeper