273: Observability: Logs, Metrics, Traces, Correlation, RED/USE, and Alerting

Learning outcomes

By the end of this lesson, you should be able to:

explain and apply structured logs in a realistic implementation;
explain and apply metrics in a realistic implementation;
explain and apply distributed tracing in a realistic implementation;
explain and apply correlation in a realistic implementation;
explain and apply dashboards in a realistic implementation.

These outcomes are practical rather than purely definitional. You should be able to choose an instrument, connect it to a question about system behavior, and use the resulting evidence while investigating a failure.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before you read, retrieve one concrete example from a previous project where the same concern appeared. Perhaps a log was the only clue to a failed request, a metric revealed a growing queue, or a trace showed that a dependency—not your own handler—was responsible for latency.

The goal is not to memorize a list of observability terms. The goal is to make a defensible decision inside a large-scale distributed service. To do that, make the service's requirements, traffic, failure modes, cost, and operational constraints explicit before choosing what to collect and retain.

Terminology

Structured logs: Emit stable, queryable fields such as timestamp, level, service, request or trace ID, principal or tenant where safe, operation, and error code. The stable shape is what makes logs useful for filtering and aggregation.
Metrics: Counters, gauges, histograms, and summaries expose trends over time. They are efficient for answering questions about rates, distributions, capacity, and saturation.
Distributed tracing: Trace context follows a request across services and records spans and dependency latency. A trace lets you inspect one request's path rather than inferring that path from unrelated records.
Correlation: Propagate request or trace IDs through HTTP, queues, jobs, and logs so one user-visible failure can be reconstructed across asynchronous boundaries.
Dashboards: Dashboards should map to service health and user journeys, not vanity infrastructure numbers. A graph is useful when it helps someone decide what to investigate or whether the system is meeting its objectives.
Alerts: Alert on actionable symptoms tied to SLO or risk, and page only when human intervention is needed. An alert that retries or autoscaling can resolve by itself is usually noise.

Mental model

Treat Observability: Logs, Metrics, Traces, Correlation, RED/USE, and Alerting as a design problem with observable inputs, outputs, invariants, and failure modes. Observability should help answer three questions: whether users are affected, which dependency is responsible, and why the behavior is occurring. Collecting every possible event without a question, a retention policy, or a plan for querying it is not observability; it is an expensive data exhaust pipe.

A strong implementation makes assumptions visible, narrows uncertainty at system boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to explain why the design is safe. For example, a request ID tells you which records belong together, while a latency histogram tells you whether a problem affects only a tail of requests or the entire population. Neither instrument replaces the other.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call. First state what must remain true. Then choose the mechanism that makes that contract observable and helps enforce or verify it. “Add tracing” is not a complete design; “identify why checkout latency exceeds its SLO across payment and queue boundaries” is a question an instrumentation plan can address.

Deep dive

1. Structured logs

When an incident occurs, a message such as request failed gives an operator very little to search for. Structured logs address that problem by emitting stable fields such as timestamp, level, service, request or trace ID, principal or tenant where safe, operation, and error code. A log consumer can then filter by operation or error code without parsing changing human prose.

Avoid secrets and uncontrolled high-cardinality payload dumps. Tokens, credentials, and sensitive customer data do not become safe merely because they are inside a log record. Large arbitrary payloads also increase storage cost and make queries less predictable. Log the identifiers and facts needed for diagnosis, subject to the service's privacy and retention requirements.

Decision rule: Use structured logs deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. For example, a stable payment_declined error code plus a safe operation field is more useful than dumping the entire payment response and hoping an operator can interpret it later.

2. Metrics

Logs are good evidence about individual events, but they are a poor substitute for a compact time series. Metrics—counters, gauges, histograms, and summaries—expose trends. A counter can show request rate or errors, a gauge can represent a current level such as queue depth, and a histogram can show the distribution of request duration instead of only an average.

The RED method focuses on request Rate, Errors, and Duration. It is useful for understanding whether a service is meeting the behavior users experience. The USE method focuses on resource Utilization, Saturation, and Errors. It is useful for finding whether a CPU, database pool, cache, queue, or other resource is approaching a limit.

Be deliberate about metric labels. A label with an unbounded value, such as a raw user ID or URL containing arbitrary input, can create a high-cardinality series explosion. Choose bounded dimensions that support diagnosis without turning the metrics system into another storage incident.

Decision rule: Use metrics deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. A request-duration metric with a defined unit and bounded route label is evidence; an unexplained number called performance is not.

3. Distributed tracing

In a distributed system, one request may pass through an API, cache, primary datastore, queue, worker, and external payment service. Trace context follows that request across services and records spans and dependency latency. This makes it possible to see where time was spent and where an error entered the path, rather than reconstructing the path from timestamps alone.

Tracing has a cost. Sampling controls how much data is recorded and retained. Errors and high-latency traces may deserve tail-based retention because they are the cases most useful during an investigation, even when ordinary requests are sampled more selectively. Sampling and retention should be chosen against the debugging questions and budget, not treated as an afterthought.

Decision rule: Use distributed tracing deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. A trace is not automatically useful because spans exist; span names, service boundaries, propagation, sampling, and sensitive-data handling all affect whether it answers the intended question.

4. Correlation

A request can cross an HTTP boundary and then continue through a queue or background job. Without propagation, the API log, event, and worker log may each be individually correct but impossible to connect. Correlation propagates request or trace IDs through HTTP, queues, jobs, and logs so one user-visible failure can be reconstructed across asynchronous boundaries.

There is a subtle boundary here: an asynchronous job may outlive the original request. Preserve the useful trace or correlation context according to the system's propagation rules, but do not assume that an ID alone explains causality or authorization. Context must also be handled safely, and it must not carry secrets or uncontrolled user input into every downstream record.

Decision rule: Use correlation deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Decide what context is created at the entry point, what is propagated, and what happens when an incoming ID is missing or malformed.

5. Dashboards

Dashboards should map to service health and user journeys, not vanity infrastructure numbers. Include latency percentiles, error rates, queue age, saturation, and critical business rates. A checkout dashboard, for example, should make it possible to see whether checkout succeeds, how long the tail takes, whether payment calls are failing, and whether queued work is aging.

Infrastructure graphs can still matter, but they should support a question. High CPU is not necessarily a user-visible incident, and low CPU does not prove that a queue or external dependency is healthy. Arrange dashboards so an operator can move from a user symptom to the responsible service or resource without having to guess which graph matters.

Decision rule: Use dashboards deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. Each panel should have a purpose, a meaningful unit or percentile, and enough context to distinguish normal variation from a real failure.

6. Alerts

Alert on actionable symptoms tied to SLO or risk, and page only when human intervention is needed. Avoid noisy per-instance alerts that autoscaling or retries already handle. A page should communicate a condition that matters to users or reliability and should lead to a concrete response.

For example, an elevated checkout error rate or an excessive queue age may warrant investigation. A single instance briefly reporting high CPU may not, especially if the service is healthy at the aggregate level and autoscaling resolves it. The exact threshold depends on the service's SLO, traffic pattern, and recovery behavior; the design should state those assumptions rather than pretending one universal number exists.

Decision rule: Use alerts deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. An alert is part of an operational contract: define the symptom, window, owner, urgency, and expected action.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Use checkout as the concrete journey. Start by writing the requirement in one sentence, list the input and output contracts, and identify which observability concept owns each failure mode. A useful requirement might be: “A valid checkout request should produce a durable order outcome, expose user-visible failures quickly, and let an operator locate latency or failure across payment and asynchronous fulfillment.”

The important move is separation. Parsing and validation belong at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Observability should follow those boundaries rather than blur them. A request log can record that validation failed, a metric can count the class of failure, and a trace can show which dependency consumed time. Mixing the concerns makes a happy-path demo look shorter, but it makes edge cases and ownership much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Walk through the diagram in order. At the edge, establish or accept the request and trace context according to the service's trust rules. At the API, validate input and record the operation and outcome without logging sensitive payloads. At the cache and datastore, measure latency and errors. When work moves to the queue, propagate the context needed to connect the user-visible request with the worker's processing, and measure queue age as well as processing duration.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate, retry, or concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem, what is recorded, and what the caller observes. For a dependency failure, distinguish the API's error from the dependency's error and identify whether retry, timeout, or queueing changes the user-visible result. This is the level of explanation expected in a senior code review or technical interview.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. Observability has its own production failure modes: missing context, inconsistent field names, unbounded labels, dropped spans, noisy alerts, and retention costs that make the data unavailable when it is needed.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. A dashboard full of data is not proof of good observability if its labels cannot be queried or its alerts cannot lead to action.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. These decisions also determine what should be logged, which errors should be counted, and where a trace should make a boundary visible.

Guided lab

Define observability for checkout: logs, RED metrics, database, cache, and queue saturation, traces across payment, correlation through asynchronous events, dashboards, and two symptom-based alerts. For each signal, name the question it answers and the cost or risk it introduces. Keep sensitive payment details out of logs and avoid labels based on arbitrary user-controlled values.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

The two alerts should describe symptoms a user or SLO would care about, not merely raw host activity. Your notes should make clear which evidence would distinguish an API problem from a payment dependency problem and which evidence would reveal queue saturation.

Edge cases and failure modes

Structured logs: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Also verify that missing or malformed correlation context does not crash logging, and that secrets or uncontrolled payloads are not emitted.
Metrics: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Check units, label bounds, counter behavior, histogram buckets, and the effect of high-cardinality values.
Distributed tracing: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Check propagation across each service and asynchronous boundary, sampling behavior, error retention, and sensitive-data handling.
Correlation: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Check what happens when a request becomes a job, when retries occur, and when several related operations run concurrently.
Dashboards: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that panels use meaningful aggregation and that a missing signal is distinguishable from a healthy zero.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.
Treating logs, metrics, or traces as interchangeable, or collecting them without deciding what question they must answer.
Adding unbounded identifiers to metric labels or logging sensitive request data because it makes an incident query seem convenient.
Paging on every low-level fluctuation instead of alerting on an actionable user or SLO symptom.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. Start with the user symptom, then compare the relevant RED signals with USE signals. Use correlation IDs to find the related records, and use a trace to locate dependency latency or an error across service boundaries. Finally, check whether the apparent absence of evidence is itself an instrumentation or sampling failure.

Interview questions

What problem do Structured logs solve, and what trade-off or failure mode would make you choose a different approach?
What problem do Metrics solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Distributed tracing solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Correlation solve, and what trade-off or failure mode would make you choose a different approach?
What problem do Dashboards solve, and what trade-off or failure mode would make you choose a different approach?

Answer each in terms of a concrete operational question, not only a definition. Be ready to discuss cost, retention, cardinality, sampling, propagation across asynchronous work, and the difference between an informative signal and an actionable page.

Checkpoint

Without notes, explain Observability: Logs, Metrics, Traces, Correlation, RED/USE, and Alerting to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.