Load balancers, and when you actually need one

A load balancer sits in front of several copies of your application and decides which copy handles each request. It is one of the most common pieces of infrastructure, and one of the most commonly added too early. This post covers what it does and, more usefully, when you actually need one.

What a load balancer does

  • Spreads traffic across several servers, so no single one is overwhelmed.
  • Checks health. It sends small test requests to each server and stops sending traffic to any that fail.
  • Gives you one address. Users talk to one stable endpoint while servers behind it come and go.
  • Often ends TLS. Many setups decrypt HTTPS at the load balancer, so the servers behind it don’t each manage certificates.

Two levels: layer 4 and layer 7

Layer 4 (transport)Layer 7 (application)
Looks atIP address and portThe HTTP request: path, headers, cookies
SpeedVery fast, little processingSlower, but much smarter
Can doForward connectionsRoute /api and /app to different services, rewrite headers, redirect
Typical useDatabases, raw TCP, very high throughputWeb apps and APIs

For a typical website or API, layer 7 is what you want. Choose layer 4 when you only need to forward connections and speed matters most.

How it picks a server

  • Round robin: take turns. Simple, and fine when requests cost about the same.
  • Least connections: send to the server with the fewest active requests. Better when some requests are slow.
  • IP hash or sticky sessions: the same user keeps landing on the same server. It works, but it hides a design problem (see below).

When you actually need one

You have a real case for a load balancer when at least one of these is true:

  1. You run more than one instance of the same service, and traffic has to be shared between them.
  2. You need to deploy without downtime. You take one instance out, update it, put it back, and repeat. Users never see an outage.
  3. One server isn’t enough, and you can’t or don’t want to make it bigger. Or a single server is a risk you can’t accept.
  4. You want automatic recovery. If an instance crashes, traffic moves to the healthy ones without anyone being paged.
  5. You split traffic by path or host. One public address in front of several different services.
  6. You want one place for HTTPS, rate limits or access rules, instead of repeating them in every service.

When you probably don’t

  • One small app on one server. A load balancer in front of a single instance adds cost and a new thing that can fail, and gives you nothing in return.
  • A static site. Put it on a CDN or static host. The traffic spreading is built in.
  • Your platform already does it. Managed platforms such as app hosts and serverless services run a load balancer for you. Adding another on top is usually a duplicate.
  • You haven’t measured a problem. If the server is at 15% CPU, the next step is not a cluster. A bigger instance, caching or a slow query fix is cheaper and simpler.

What it costs you

  • It can become the single point of failure. If the load balancer goes down, everything behind it is unreachable. Managed cloud load balancers are built redundant for this reason, but a hand-run single one is not.
  • More moving parts. Health checks, timeouts and draining connections all need to be configured correctly. A health check that is too strict can take healthy servers out.
  • Your app has to be stateless. If a user’s login or cart lives in one server’s memory, the next request may land on a different server and lose it. Sticky sessions paper over this. The better fix is to keep state in a shared store such as a database or cache, so any server can serve any request.
  • Money. Managed load balancers bill for the hours they run and the traffic they handle.

A quick checklist

Ask these in order. Stop at the first “no”.

  1. Do I have, or plan to have, more than one instance of this service?
  2. Is my app stateless, or can I make it so?
  3. Is my platform not already providing a load balancer?
  4. Do I need zero-downtime deploys, automatic recovery or more capacity than one server gives?

If all four are “yes”, add a load balancer. If you stopped at the first question, you likely don’t need one yet.

Takeaway

A load balancer solves a specific problem: several servers, one front door. Add it when you have that problem. Before then, a single well-sized server with good monitoring is simpler to run, cheaper, and easier to debug.

TODO: add a diagram of your own set-up and any numbers from real projects.

Nikitha
Nikitha
Senior Software Engineer

TODO One or two sentences describing your work and interests.