251: DNS, Naming, Service Discovery, and Global Traffic Routing

Learning outcomes

By the end of this lesson, you can:

explain and apply DNS hierarchy in a realistic implementation;
explain and apply TTL and caching in a realistic implementation;
explain and apply health and geo routing in a realistic implementation;
explain and apply internal discovery in a realistic implementation;
explain and apply stable names in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before you read, retrieve one concrete example from an earlier project where this concern appeared. Perhaps a frontend called an API through a hostname, a service used a database endpoint instead of a database IP, or a deployment replaced instances without changing the client configuration. The point is not to memorize vocabulary. It is to connect the vocabulary to a decision in a large-scale distributed service, where requirements, traffic, failure modes, cost, and operational constraints all need to be stated explicitly.

Terminology

DNS hierarchy: Resolvers walk through the DNS hierarchy and cache records obtained from authoritative nameservers.
TTL and caching: A DNS TTL tells resolvers how long an answer may be cached, but it cannot force every client or intermediate cache to refresh immediately.
Health and geo routing: Authoritative DNS or a global traffic manager can select an endpoint by geography, latency, weighted policy, or health. Failover speed still depends on both detection and the answers that are already cached.
Internal discovery: Microservices can find instances through platform DNS, a service registry, or a service mesh. Each mechanism has its own health, consistency, and rollout behavior.
Stable names: Clients should depend on stable service names rather than instance IP addresses. The instances behind the name may be replaced, scaled, or moved.
Failure modes: DNS provider outages, bad records, expired domains or certificates, cache-poisoning defenses, and split-brain internal records all belong in the operational risk analysis.

Mental model

Treat DNS, Naming, Service Discovery, and Global Traffic Routing as a design problem with observable inputs, outputs, invariants, and failure modes. A name decouples a client from a changing address, but that decoupling is not instantaneous: DNS caches, TTLs, health checks, and internal discovery can all delay the visibility of a change. A sound implementation makes its assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe.

A useful sequence for both interviews and production work is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement straight to a library call or a provider feature. First state what must remain true. Then select the mechanism that enforces that invariant and decide how you will observe when it stops being true.

Deep dive

1. DNS hierarchy

When a client uses a public hostname, it does not normally receive an answer from one magical global table. A resolver follows the DNS hierarchy, beginning with the root and moving through the relevant top-level and authoritative nameservers, then caches the result. The record may be an A or AAAA record, a CNAME, an alias-like provider record, or a record controlled by a routing policy.

The hierarchy gives ownership boundaries and lets operators change the authoritative answer without reconfiguring every client. It also introduces dependencies: delegation, nameserver availability, record configuration, and the resolver's cached view all affect what a client sees. DNS is not HTTP, so a successful lookup only identifies an endpoint; it does not prove that the application is healthy or that a request will return 200.

Decision rule: Use DNS hierarchy deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

2. TTL and caching

The TTL on a DNS record controls how long a compliant resolver may reuse an answer before asking again. A low TTL can make planned address changes or routing changes visible sooner, but it increases query volume and does not eliminate stale answers. Clients, recursive resolvers, operating-system caches, libraries, and provider-specific behavior can all affect when a change is observed.

That is why “set the TTL to zero and fail over instantly” is not a complete availability plan. During an incident, some clients can continue using an old endpoint while others receive the new one. A deployment or migration must tolerate that overlap. DNS caching also differs from an HTTP cache: DNS selects an address, while HTTP caching concerns responses, freshness, and revalidation.

Decision rule: Use TTL and caching deliberately when they make the contract or invariant easier to prove. If a short TTL only creates query load while the system still cannot tolerate stale answers, make that limitation explicit instead of treating the TTL as a guarantee.

3. Health and geo routing

Authoritative DNS or a global traffic manager can choose among endpoints using geography, measured latency, weighted distribution, or health status. These policies answer different questions. Geo routing may keep a user near a region, latency routing may optimize observed network distance, weighted routing may support a canary, and health routing may remove an endpoint that fails a check.

Health routing is not the same as application correctness. A shallow check may report that a process is listening while the dependency it needs is unavailable; a deep check may remove too much capacity when a noncritical dependency is degraded. Detection also takes time, and cached DNS answers can keep sending clients to an endpoint after the routing system has marked it unhealthy. State the recovery target in terms of detection time, cache behavior, and client retry behavior rather than promising instant failover.

Decision rule: Use health and geo routing deliberately when they make the contract or invariant easier to prove. If the policy hides a trade-off—such as locality versus failover capacity—make the trade-off part of the design rather than relying on the provider's default.

4. Internal discovery

Inside a distributed system, service instances are often ephemeral. Microservices can discover them through platform DNS, a service registry, or a service mesh. Platform DNS may provide a stable service name; a registry may track instance metadata and health; a mesh may provide discovery, load balancing, retries, and telemetry through sidecars or node-level proxies.

The discovery system is itself a distributed dependency. It needs health semantics, consistency expectations, and rollout behavior. Ask what happens when a registry is temporarily unavailable, when an instance is registered before it is ready, when deregistration is delayed, or when different callers receive different membership views. A client also needs a policy for no endpoints, stale endpoints, connection timeouts, and retries; naming alone does not solve those cases.

Decision rule: Use internal discovery deliberately when it makes the contract or invariant easier to prove. If it only hides instance management without defining health and consistency behavior, prefer the more explicit design.

5. Stable names

Clients should depend on stable service names rather than instance IPs. An instance may be replaced during a deploy, moved to another host, or added and removed as traffic changes. A stable name lets the platform or discovery layer update the membership while the client keeps the same contract.

Stable does not mean permanent or universally reachable. The name still has a scope, an owner, an authentication policy, and failure behavior. Define whether it is public or internal, whether it resolves to one endpoint or many, and how callers handle an empty or stale result. This separation is what lets a deployment change instances without forcing every caller to learn new IP addresses.

Decision rule: Use stable names deliberately when they make the contract or invariant easier to prove. If a name conceals a dependency on a particular region, protocol, or availability level, document that boundary instead of treating the name as a complete abstraction.

6. Failure modes

DNS provider outages, misconfiguration, expired domains or certificates, cache-poisoning defenses, and split-brain internal records belong in operational risk analysis. A domain can be registered correctly while its records are wrong. A hostname can resolve correctly while its certificate is expired or does not cover that hostname. An internal registry can be reachable while serving an inconsistent or stale membership view.

For each failure, identify the owner, the detection signal, the client-visible symptom, and the recovery path. Also distinguish a DNS failure from an HTTP 404: in the first case the client may never reach the application; in the second, the application or edge returned an HTTP response. That distinction changes what you inspect and which team can fix the problem.

Decision rule: Analyze failure modes deliberately when they make the contract or invariant easier to prove. If the analysis only lists provider features without describing detection, cached state, and recovery, it is not yet an operational design.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start with a one-sentence requirement, then write the input and output contracts and identify which concept above owns each failure mode. The useful separation is by responsibility: parsing and validation belong at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing those concerns can make a happy-path demo look shorter, but it makes stale answers, retries, partial failure, and other edge cases much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Read the diagram as a chain of contracts, not as a promise that every request takes exactly this path. The client resolves a name, the edge and load balancer select a reachable service instance, the API may use a cache or primary datastore, and asynchronous work can continue through a queue or stream. Each boundary can fail independently and can have different timeout, retry, and consistency behavior.

Walk through at least four cases: the normal path; an empty or missing value; a duplicate, retry, or concurrent path where relevant; and a dependency failure. For every case, state which layer detects the problem, what evidence you would inspect, and what the caller observes. For example, a stale DNS answer may be invisible to the authoritative service but visible in resolver behavior, while a datastore outage may be reported by the API as a structured dependency error. This is the level of explanation expected in a senior code review or technical interview.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high-cardinality traffic. For naming and routing, include cached answers, resolver behavior, health-check intervals, certificate renewal, regional capacity, and the possibility that different clients see different endpoints at the same time. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and that network input is untrusted. In particular, resolving a trusted-looking name is not authorization; the service still needs to authenticate and authorize the request.

Guided lab

Design public and internal naming for a multi-region API. Choose TTL and routing behavior, show how an instance deploy changes without clients learning new IPs, and describe the limitations of regional failover. Your design should say who owns the public zone, how internal callers discover the service, what health check is authoritative, and how long stale answers may remain useful or harmful.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing. Check answers at the resolver, routing, service, and dependency boundaries where appropriate.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

Edge cases and failure modes

DNS hierarchy: Test absence, malformed input, duplicates, ordering or concurrency where applicable, and behavior at the smallest and largest credible sizes. Include a delegation or authoritative-record mistake and distinguish lookup failure from an application response.
TTL and caching: Test absence, malformed input, duplicates, ordering or concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that stale answers are possible and that the design remains safe during the overlap between old and new endpoints.
Health and geo routing: Test absence, malformed input, duplicates, ordering or concurrency where applicable, and behavior at the smallest and largest credible sizes. Check detection delay, cached answers, uneven regional capacity, and the difference between a shallow and a deep health check.
Internal discovery: Test absence, malformed input, duplicates, ordering or concurrency where applicable, and behavior at the smallest and largest credible sizes. Include no endpoints, stale membership, delayed deregistration, and inconsistent views during rollout.
Stable names: Test absence, malformed input, duplicates, ordering or concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that instance replacement does not require client configuration changes and that the name's scope and failure contract are clear.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case and record the exact hostname, resolver or client context, timestamp, and region. Inspect the actual DNS answer and its TTL, then check whether the answer came from a local or recursive cache. If the name resolves, inspect the edge or load-balancer decision, health-check status, certificate, service logs, and dependency response. Trace the boundary where the invariant first becomes false and fix the owning layer rather than adding a downstream patch. A DNS failure, a stale DNS answer, a TLS failure, an HTTP error, and an application dependency error are different symptoms with different owners.

Interview questions

What problem does DNS hierarchy solve, and what trade-off or failure mode would make you choose a different approach?
What problem does TTL and caching solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Health and geo routing solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Internal discovery solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Stable names solve, and what trade-off or failure mode would make you choose a different approach?

Checkpoint

Without notes, explain DNS, Naming, Service Discovery, and Global Traffic Routing to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code. Be specific about what a caller observes when the name is stale, when no healthy instance is available, and when the application is reachable but returns an error.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.