262: Queues, Message Brokers, Pub/Sub, and Streams

Learning outcomes

By the end of this lesson, you can:

explain and apply work queues in a realistic implementation;
explain and apply publish/subscribe in a realistic implementation;
explain and apply streams/logs in a realistic implementation;
explain and apply at-least-once delivery in a realistic implementation;
explain and apply ordering in a realistic implementation.

These outcomes are intentionally practical. You should be able to connect each term to a system requirement, choose an appropriate delivery model, and explain what happens when the system retries, falls behind, receives a duplicate, or loses a dependency.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where the same concern appeared. Perhaps a request had to trigger slow background work, several parts of the system needed to react to one event, or a consumer had to catch up after being unavailable. The specific technology is less important than the design question it exposed.

The goal is not to memorize messaging terminology. It is to make a defensible decision inside a large-scale distributed service. That decision needs explicit requirements, traffic expectations, failure modes, cost, and operational constraints. Without those details, choosing a queue, broker, pub/sub system, or stream often becomes a product-name exercise rather than engineering.

Terminology

Work queues: A queue distributes jobs among workers; a message is typically handled by one consumer group member. This is useful when one piece of work should be completed once by one member of a worker pool.
Publish/subscribe: Pub/sub fans one event to multiple independent subscribers. Each subscriber can process the same event for a different purpose without the producer knowing the subscriber's implementation.
Streams/logs: Append-only logs retain ordered records for replay, multiple consumer groups, and event-driven integrations. Unlike a simple handoff queue, a log's retained history can be read again by a consumer that needs to rebuild state or recover from an outage.
At-least-once delivery: Retries can deliver the same message more than once, so consumers should be idempotent or deduplicate using stable event/job identity. Acknowledging work only after processing reduces loss, but it leaves a window in which a completed message can be delivered again.
Ordering: Global ordering is expensive and rarely necessary. Most domains need ordering only within a meaningful scope, such as one account, order, or partition.
Backlog and dead letters: Queue depth and message age are saturation signals. A dead-letter or quarantine path isolates messages that cannot be processed successfully after bounded retries. Treat backlog and dead letters as precise engineering concepts, not merely vocabulary.

Mental model

Treat Queues, Message Brokers, Pub/Sub, and Streams as a design problem with observable inputs, outputs, invariants, and failure modes. Asynchronous messaging decouples producer latency from consumer latency and lets producers continue when consumers are temporarily unavailable. That decoupling is valuable, but it does not remove complexity. It introduces delivery semantics, ordering decisions, duplicates, backlog, and operational state that must be designed explicitly.

A strong implementation makes its assumptions visible, narrows uncertainty at system boundaries, and leaves enough evidence to explain why the design is safe. That evidence might be tests, types, constraints, metrics, logs, dashboards, or diagrams. For example, “this event is processed once” is not a sufficient design statement when a retry can occur. The useful statement is closer to “processing may occur more than once, but the effect is idempotent for a stable event ID, and the queue is acknowledged only after the effect is durable.”

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call. First state what must remain true. Then choose the mechanism that enforces it. For instance, “send an email eventually” may need a work queue, while “let billing, analytics, and inventory independently observe an order event” points toward pub/sub or a retained stream. The distinction comes from the contract, not from the label on the infrastructure product.

Deep dive

1. Work queues

A queue distributes jobs among workers; a message is typically handled by one consumer group member. Use it for background work such as emails, image processing, and reconciliation. The producer places a job in the queue and does not wait for the worker's full processing time. Workers claim jobs, perform the work, and acknowledge them according to the queue's delivery contract.

This model is a good fit when the work is a task rather than a fact that many independent systems must observe. It also gives a worker pool a way to absorb bursts: producers can continue to enqueue while workers drain the backlog at a controlled rate. That buffer is not free, however. The system must define acceptable queue age, retry behavior, and what happens when a job cannot ever succeed.

Decision rule: Use work queues deliberately when they make the contract or invariant easier to prove. If a queue only reduces typing while hiding an assumption, prefer the more explicit design. Ask who owns a job, whether more than one consumer needs it, when it is considered complete, and how a duplicate affects the result.

Pub/sub fans one event to multiple independent subscribers. An order-created event, for example, might be useful to payment, email, analytics, and inventory. Those subscribers should not have to coordinate through one worker queue, because each has a separate responsibility and separate retry or availability profile.

Each subscriber may need its own durable cursor or retry state if missing events is unacceptable. One slow subscriber should not silently determine whether every other subscriber receives the event. The producer publishes the event once, while subscribers independently track their progress and handle failures.

Decision rule: Use publish/subscribe deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Be precise about whether subscribers receive a durable event, how long it is retained, and whether a new subscriber can process historical events.

3. Streams/logs

Append-only logs retain ordered records for replay, multiple consumer groups, and event-driven integrations. A consumer reads records without removing the history for other consumers. This makes a stream useful when different applications need their own view of the same event history or when rebuilding derived state after a bug or outage matters.

Partitioning determines parallelism and ordering scope. Records in one partition can have a defined sequence, while different partitions can be processed concurrently and may not have a meaningful global order. The partition key therefore carries domain meaning: choosing an order ID can preserve the order's events, while choosing a random key may maximize distribution but lose that guarantee.

Retention is also part of the design. A retained log supports replay only for the period and storage boundary the system actually keeps. Replay can repeat side effects, so consumers still need idempotency and a clear distinction between rebuilding internal state and sending an external notification again.

Decision rule: Use streams/logs deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. State the required retention period, consumer groups, partition key, replay behavior, and ordering scope before selecting the implementation.

4. At-least-once delivery

Retries can deliver the same message more than once. A worker may finish the side effect and crash before its acknowledgment reaches the broker; the broker then reasonably makes the message available again. This is why consumers should be idempotent or deduplicate using stable event or job identity.

Idempotency means that applying the same logical operation again does not produce an additional unintended effect. A payment consumer might record a processed event ID under a uniqueness constraint before allowing the operation to be treated as new. The exact mechanism depends on the side effect, but a random attempt ID that changes on every retry will not provide deduplication.

At-least-once delivery favors avoiding silent loss, but it does not promise exactly-once business results across arbitrary external systems. The consumer must define what it can safely repeat, what must be guarded by a transaction or constraint, and what requires reconciliation.

Decision rule: Use at-least-once delivery deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Document the acknowledgment point, stable identity, idempotency boundary, retry limit, and recovery path.

5. Ordering

Global ordering is expensive and rarely necessary. A single ordered sequence limits parallelism and can make one slow or failed item block unrelated work. Preserve order per entity or partition when the domain requires it, and design around out-of-order delivery elsewhere.

The first question is not “How do I force every message into one order?” It is “Which messages must be ordered relative to one another?” A user's profile changes may need per-user ordering, while events for unrelated users can proceed concurrently. If a consumer receives an event whose predecessor has not arrived, it needs an explicit policy: wait, buffer with a bound, reject for retry, or reconcile from authoritative state.

Decision rule: Use ordering deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. State the ordering scope, partition key, behavior for late events, and cost of blocking before treating global order as a requirement.

6. Backlog and dead letters

Queue depth and message age are saturation signals. Depth tells you how much work is waiting; age tells you how long the oldest work has waited. A queue can have a modest depth and still violate an SLA if workers are slow, or a large depth and remain healthy if the workload is intentionally batch-oriented. Monitor the signal that matches the user or business requirement.

Poison messages need bounded retries and a dead-letter or quarantine workflow with alerting and replay tools. Retrying malformed input forever consumes worker capacity and can prevent healthy messages from being processed. A quarantine path should preserve enough context to diagnose the failure, while replay should be deliberate and safe for an idempotent consumer. Operators also need to know whether the failure is in the message, the consumer code, a dependency, or configuration.

Decision rule: Use backlog and dead letters deliberately when they make the contract or invariant easier to prove. If they only reduce typing while hiding an assumption, prefer the more explicit design. Define thresholds, retry backoff, retention, alert ownership, inspection access, and the conditions under which replay is allowed.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be explicit. Start by writing the requirement in one sentence, list the input and output contracts, and identify which concept above owns each failure mode. For example, “when an order is created, each downstream capability must eventually process a valid order-created event, with no duplicate business effect” is more useful than “add messaging.”

The important move is separation. Parsing or validation belongs at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes retries, malformed messages, and edge cases much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                         |          |
                         +------> Primary datastore
                         |
                         +------> Queue / Stream -> Workers

Read the diagram as a set of boundaries, not as a claim that every system needs every component. The API receives the request, the datastore holds authoritative state, and the queue or stream separates request handling from asynchronous work. A cache may reduce reads but is not automatically the source of truth. The messaging layer also does not make invalid input valid or guarantee that a worker's side effect is performed exactly once.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate/retry/concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem and what the caller observes. For the duplicate order event, for example, the consumer should use the stable event identity or a database constraint rather than assuming the broker will never redeliver. For a dependency failure, state whether the message is retried, delayed, quarantined, or reported as a failure to the caller.

This is the level of explanation expected in a senior code review or technical interview: name the invariant, identify its owner, describe the observable failure, and explain how the system recovers.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. A consumer that works in isolation may still fail during a rolling deploy if old and new message schemas overlap, or during a traffic spike if backlog age is not monitored.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. For messaging, that evidence commonly includes publish rate, processing rate, queue depth, oldest-message age, retry counts, dead-letter counts, processing latency, and duplicate rate.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. A message arriving from a trusted-looking topic still needs schema and authorization decisions appropriate to the system.

Guided lab

Design an order-created event flow to payment, email, analytics, and inventory consumers. Specify topic or queue ownership, the partition key, retry and idempotency behavior, the dead-letter queue (DLQ), and what can be replayed. Do not stop at naming a broker. Explain why each consumer receives the event, what state it owns, and which effects are safe to repeat.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

The lab should make the trade-offs visible. For example, a shared work queue might be appropriate for one owned task, but it would be the wrong model if payment, email, analytics, and inventory must each receive the event independently. A stream may make replay easier, but it also makes retention, partitioning, and repeated side effects part of the operational contract.

Edge cases and failure modes

Work queues: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Publish/subscribe: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Streams/logs: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
At-least-once delivery: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Ordering: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.

These cases should be interpreted against the contract. “Absence” can mean a missing required field, no available message, or a consumer that starts with no retained history. “Largest credible size” includes payload size, backlog, concurrency, and event rate where those dimensions affect the design. The point is to expose what the implementation assumes before production exposes it for you.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. In a messaging system, inspect the message ID, partition or routing key, attempt count, timestamps, acknowledgment state, consumer offset or cursor, and the relevant worker logs. Compare publish rate with processing rate, then check whether the failure is deterministic or dependency-related.

If a message keeps returning, determine whether the handler fails before acknowledgment, the acknowledgment is lost, or the handler completes a side effect and then crashes. If the queue grows, determine whether producers accelerated, consumers slowed down, a dependency is timing out, or poison messages are consuming the retry budget. If events appear out of order, inspect the ordering scope and partition key before assuming the broker is broken.

Interview questions

What problem do Work queues solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Publish/subscribe solve, and what trade-off or failure mode would make you choose a different approach?
What problem do Streams/logs solve, and what trade-off or failure mode would make you choose a different approach?
What problem does At-least-once delivery solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Ordering solve, and what trade-off or failure mode would make you choose a different approach?

Answer each question in terms of a requirement, not a favorite tool. Include the delivery and ordering contract, the failure mode that matters, and the operational cost of the alternative.

Checkpoint

Without notes, explain Queues, Message Brokers, Pub/Sub, and Streams to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

If your explanation says only that a queue is “for asynchronous work,” it is not finished. Say who receives a message, whether the message is retained, what happens on retry, which ordering scope is guaranteed, and how an operator detects a stuck or poisonous message.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.