Service Discovery
"Where is the payments service right now?" When instances come and go every few minutes, that's a real question.
The story
The orders service has the payments service's address in its config file: three IP addresses, one per instance. It works for a year because the payments servers never change.
Then payments moves to autoscaling. Instances now start and stop all day and get new IPs each time. The day of the move, the config still lists the three original IPs. Two of those instances have already been replaced, so two-thirds of payment calls go to addresses where nothing is listening, and checkouts fail until someone edits the config and redeploys orders.
The team switches to a DNS name, payments.internal, updated as instances change. That's better, until a deploy replaces every instance and some clients keep using the old addresses for the next 20 minutes. Their HTTP library cached the DNS answer far longer than its TTL.
Both failures have the same cause: the caller's idea of where the service lives was out of date. Service discovery is the system that keeps that idea current.
The idea
Every discovery system has three parts:
- Registration: instances announce themselves when they start and are removed when they stop. Either the instance registers itself, or the platform does it (Kubernetes, ECS, a Consul agent).
- Health: the registry only returns instances that pass health checks or keep renewing a heartbeat/TTL.
- Lookup: callers get the current list, by asking or by being pushed updates.
Client-side vs server-side discovery
| Client-side | Server-side | |
|---|---|---|
| Examples | Netflix Eureka + Ribbon, gRPC with xDS, Consul client libraries | Kubernetes Services, AWS ALB/Cloud Map, Consul + a proxy |
| Hops | None extra | One extra hop through the proxy |
| Caller complexity | Every language needs the library | Callers just use a name |
| Smart balancing | Yes, in the client | In the proxy |
Most teams today use the server-side form without noticing, because the platform provides it.
How Kubernetes does it
The name and the virtual IP never change. Behind them, the list of real pod IPs is updated as pods start, stop and pass or fail their readiness probes. That fixes both failures in the story: callers never hold pod IPs, and the cached DNS answer (the ClusterIP) stays valid.
A service mesh (Istio, Linkerd) goes one step further: a sidecar proxy next to every pod gets live endpoint lists from a control plane and adds retries, mTLS and per-request load balancing.
What goes wrong
- Stale caches. Clients that cache DNS or registry results past their TTL call dead instances. Respect TTLs, keep them short (5–30s), and retry on connection failure.
- Registering before ready. An instance registered before its caches and connections are warm gets traffic it can't serve. Separate liveness (restart me) from readiness (send me traffic).
- Deregistering too late. On shutdown: deregister first, wait for callers to notice, finish in-flight requests, then exit. Kubernetes' preStop hook plus a grace period exists for this.
- The registry as a single point of failure. Run it as a cluster (Consul and etcd use Raft, see Leader Election), and let clients keep using their last known list if it's briefly unreachable. Eureka deliberately favours availability here.
Cheat sheet
| Question | Answer |
|---|---|
| Hard-coded IPs | Only for things that truly never move. Never for autoscaled services. |
| On Kubernetes | Use Services + cluster DNS. Readiness probes decide who gets traffic. |
| Outside Kubernetes | Consul / Cloud Map / Eureka, or a load balancer per service |
| Client-side vs server-side | Server-side by default. Client-side when you need smart per-request balancing and control the client languages. |
| DNS TTL | 5–30s, and make sure the client actually respects it |
| Health | Readiness (traffic) ≠ liveness (restart) |
| Shutdown | Deregister → drain → exit |
| Registry down | Clients keep the last known good list |
| Mesh | When you also want mTLS, retries and traffic shifting across many services |
Say it in the interview
"On Kubernetes, services call each other by Service name. DNS returns a stable ClusterIP, and kube-proxy routes to pods that are passing readiness probes, so callers never hold instance IPs. On shutdown a preStop hook gives the endpoint list time to drop the pod before it stops taking requests. If we needed mTLS and smarter balancing across many services, I'd add a mesh."
Go deeper
- SD 3 · Q48 — Service Discovery: the chain of whys.
- API Gateway: discovery for traffic coming from outside.
- Load Balancing: health checks and draining in depth.