Load balancing and horizontal scale

Vertical first, honestly

Vertical scaling — a bigger machine — is underrated. It requires no code changes, no distributed-systems bugs, and modern hardware is enormous. A single server with 64 cores and 256 GB of RAM handles more traffic than most products will ever see.

Go horizontal when you hit one of three walls: the biggest available machine is not enough, you need redundancy because one machine is one outage, or you need presence in multiple regions.

What "stateless" actually requires

Horizontal scaling only works if any request can go to any instance. That means nothing that must survive between requests lives in the process:

• Sessions → Redis or a signed cookie, not in-process memory.
• Uploaded files → object storage, not the local disk.
• In-memory caches → fine as a performance layer, but every instance has its own, so hit rates drop and invalidation is per-instance. Shared Redis for anything that must be consistent.
• Scheduled jobs → run on one instance, not all of them, or you will send every email five times. Use a leader lock or a dedicated scheduler.
• WebSockets → connections are pinned to one instance, so broadcasting needs a pub/sub layer.

That last-mile list is where "we'll just add another instance" usually goes wrong.

Stateless instances behind a load balancer, with reads split off to replicas.

Load balancing

Round-robin is the default and is usually fine. Least-connections is better when request durations vary a lot. Consistent hashing matters when instances hold local state — it minimises how many keys move when an instance joins or leaves, which is why it is the backbone of sharding and distributed caches.

Two things the load balancer must do properly: health checks that reflect real readiness (can it reach the database?), and graceful shutdown — stop accepting new connections, drain in-flight requests, then exit. Without draining, every deploy drops requests.

Scaling the database

The application tier is the easy part. The database is where scaling actually gets decided.

Read replicas come first: writes go to the primary, reads to replicas. This solves read-heavy load with almost no application change — except replication lag. A user who writes then immediately reads may hit a replica that has not caught up. Route reads to the primary for a short window after a write.

Sharding comes last, and it is a genuinely large step. You split data across independent databases by a shard key. The consequences are permanent:

• Cross-shard joins and cross-shard transactions are gone. You do them in application code, or you avoid them.
• The shard key choice is close to irreversible. Pick one with even distribution and one that most queries filter on. Sharding by user_id is common because most queries are per-user.
• Rebalancing is an operational project. Consistent hashing or virtual buckets make it survivable.

Exhaust the alternatives first: better indexes, caching, read replicas, archiving cold rows, moving analytics off the primary. Most teams that shard did not need to.

How the decisions cascade

This is the part interviews probe. Add instances → sessions must move out. Add replicas → replication lag becomes a product decision. Shard → joins disappear, so your data model changes. Multi-region → CAP is now your daily problem. Each step forces the next, which is why "just add a server" is never the whole answer.

Prove you know it

Take something you have deployed and answer: if I run three copies right now, what breaks? Name every piece of per-instance state. Then fix the top one.