268: Distributed Transactions, Sagas, Transactional Outbox, CDC, and Compensation

Learning outcomes

By the end of this lesson, you can:

explain and apply local transactions in a realistic implementation;
explain and apply saga in a realistic implementation;
explain and apply orchestration and choreography in a realistic implementation;
explain and apply transactional outbox in a realistic implementation;
explain and apply change data capture in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the 01–06 foundation and the preceding lessons in this module. Before you begin, retrieve one concrete example from an earlier project where a single operation crossed more than one boundary: perhaps an order update that also triggered a notification, or a write that had to reach a search index. You are not trying to memorize a collection of distributed-systems terms. You are trying to make a decision you can defend for a large-scale distributed service.

That decision needs explicit requirements, traffic expectations, failure modes, cost limits, and operational constraints. Those details determine whether a local transaction is enough, whether a workflow needs a saga, and whether an outbox or CDC pipeline is appropriate.

Terminology

Local transactions: Keep each service’s invariants atomic within its own datastore. A local transaction can commit or roll back the changes covered by that datastore, but it does not automatically include another service’s database.
Saga: A saga is a sequence of local transactions connected by messages or events, with compensating actions for failures discovered later. Compensation changes business state; it is not a delayed database rollback.
Orchestration and choreography: An orchestrator owns workflow state and issues commands for the next step. In choreography, services react to events and decide their next action without a single coordinator. The choice affects visibility, coupling, and where workflow decisions live.
Transactional outbox: Write domain state and an outbox event in the same local transaction. A separate publisher reads the outbox and publishes asynchronously, so a process crash cannot commit the state change while losing the corresponding event.
Change data capture: CDC streams changes from a database log to downstream consumers. It can feed integration, search, or analytics pipelines, but the team still has to control schema evolution, ordering assumptions, and the meaning of the emitted events.
Compensation limits: Some effects cannot be truly undone. An email that was sent or a package that was shipped can only be followed by corrective action, such as sending a correction or issuing a return.

Mental model

Treat Distributed Transactions, Sagas, Transactional Outbox, CDC, and Compensation as a design problem with observable inputs, outputs, invariants, and failure modes. When a business workflow spans services that commit independently, a single atomic rollback is usually unavailable. The design must therefore record progress, tolerate retries, and define what happens when a later step fails.

A strong implementation makes its assumptions visible. It narrows uncertainty at service boundaries and leaves evidence, such as tests, types, database constraints, metrics, logs, or diagrams, that explains why the design is safe. “The message was probably delivered” is not an invariant; a durable state transition and an observable delivery attempt are evidence you can inspect.

A useful sequence for both an interview and a production design is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement straight to a library call. First state what must remain true. Then identify which service owns that invariant, how progress is recorded, and which mechanism makes the guarantee enforceable.

Deep dive

1. Local transactions

The first question is whether the invariant belongs entirely to one service and its datastore. If it does, keep the related writes in a local transaction. For example, creating an order and reserving the order’s own line-item records can be one atomic database operation when those records have the same owner.

Do not hold that database transaction open while waiting for a slow network call to payment, inventory, or shipping. A network dependency can time out, retry, or remain unavailable, unnecessarily retaining locks and connections. Commit the local fact first, then coordinate the cross-service work explicitly.

Decision rule: Use a local transaction when it makes the service’s contract or invariant easier to prove. If it merely saves a few lines while hiding a cross-service assumption, prefer the more explicit design.

2. Saga

Suppose an order is created, payment succeeds, inventory is reserved, and shipping later rejects the request. There is no safe database rollback that reaches all four services after the fact. A saga models this as a sequence of local commits and defines a business action for the failure path, such as releasing inventory or refunding payment.

The distinction matters: compensation is a new operation with its own retries, failures, permissions, and audit trail. It may restore a business outcome, but it cannot guarantee that the world looks exactly as if the original steps never happened. A refund may take time, and a notification may already have been delivered.

Decision rule: Use a saga when a workflow spans independently committed services and eventual completion is acceptable. Define the state transitions, retry policy, idempotency behavior, and terminal failure states rather than treating compensation as magical rollback.

3. Orchestration and choreography

An orchestrated saga has a coordinator that owns workflow state and sends commands such as AuthorizePayment, ReserveInventory, and CreateShipment. This makes the order of steps, timeout handling, and compensation policy visible in one place. The coordinator becomes another production component, however, and must be made durable, observable, and highly available.

In a choreographed saga, a service publishes an event such as OrderCreated; payment, inventory, and shipping services subscribe and react. This can reduce direct coupling to a coordinator, but the workflow is spread across event handlers. It becomes harder to answer “what happens next?”, detect a stalled order, or change the process without understanding several consumers.

The useful distinction is explicit control versus distributed autonomy, not “good architecture” versus “bad architecture.” Orchestration is often easier to audit for a long or highly regulated workflow. Choreography can fit a small number of stable reactions, but it needs strong event contracts, correlation identifiers, observability, and safeguards against event cycles.

Decision rule: Choose orchestration and choreography based on where workflow ownership, visibility, and change are easiest to manage. Whichever model you choose, make retries, duplicate delivery, ordering assumptions, timeouts, and compensation behavior explicit.

4. Transactional outbox

The classic failure is simple: the service commits an order to its database, then crashes before publishing OrderCreated. A retry can create the opposite problem if it publishes first and the database transaction fails. A transactional outbox closes this local gap by storing the business change and an outbox record in the same database transaction.

A publisher later reads unpublished outbox records and sends them to the queue or stream. The publisher may crash after publishing but before marking the record as sent, so consumers must expect at-least-once delivery and handle duplicates idempotently. The outbox also needs retention, retry and backoff rules, visibility into stuck records, and a policy for poison messages.

Decision rule: Use a transactional outbox when a local state change must reliably produce an asynchronous message. It does not provide a global transaction, exactly-once processing, or automatic consumer correctness; it provides a durable handoff from the local transaction to the publisher.

5. Change data capture

CDC reads database log changes and streams them to consumers without requiring every application write path to publish an event directly. That makes it useful for synchronizing a search index, loading analytics data, or integrating an existing system whose database is the available source of change.

The database record is not automatically a well-designed domain event. A row update may expose storage details, omit the business reason for the change, or change shape when a schema migration occurs. Consumers need a versioned contract, a clear interpretation of deletes and updates, ordering and replay expectations, and a plan for incompatible changes.

Decision rule: Use CDC when log-based replication is the right integration boundary and downstream consumers can tolerate its semantics. Prefer intentional domain events when consumers need business meaning that cannot be recovered reliably from row-level changes. In either case, monitor lag, failures, replay behavior, and schema compatibility.

6. Compensation limits

Compensation works best when a previous action has a well-defined inverse, such as releasing a reservation or issuing a refund. It becomes less exact when the action has external or irreversible effects. Sending an email cannot be unsent, and shipping a package cannot be reversed by changing a database row.

Model these realities in the state machine. A workflow may need states such as payment_refund_pending, manual_review_required, or shipped_correction_required, rather than pretending that every failure returns the order to created. Record the original action, the compensation attempt, its result, and the human or automated follow-up required.

Decision rule: Use compensation limits as a design constraint. Identify irreversible effects before implementation and make their corrective actions, user-visible states, and operational ownership explicit.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be explicit. Start with one sentence describing the requirement. Then list the input and output contracts and assign each failure mode to the concept that owns it. For a checkout flow, the order service may own order state, payment may own authorization and refund, inventory may own reservation, and shipping may own fulfillment.

The important design move is separation of concerns. Parsing and boundary validation belong at the boundary. Domain rules belong in the domain or service layer. Persistence rules belong in the database or repository. Presentation rules belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes retries, partial completion, and edge cases much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                         |          |
                         +------> Primary datastore
                         |
                         +------> Queue / Stream -> Workers

For each request, ask what is synchronous and what is eventual. An API instance might commit the order and outbox record locally, then return a pending status while workers coordinate payment, inventory, and shipping. Correlation IDs connect those messages and state transitions so an operator can reconstruct one checkout across services. A cache can improve reads, but it does not replace the primary invariant or make a failed cross-service write atomic.

Walk the example through at least four cases:

Normal path: the order is committed, the event is published, each service completes, and the final status is visible to the client.
Empty or missing value: identify whether the request is rejected at the boundary or whether an optional field has a defined default. Do not let malformed input become a confusing downstream failure.
Duplicate, retry, or concurrent path: use an idempotency key, unique constraint, event ID, or state transition guard where appropriate. State what happens if the same payment command or event is processed twice.
Dependency failure: define the timeout, retry limit, compensation, and terminal state when payment, inventory, the queue, or a worker is unavailable.

For every case, state which layer detects the problem and what the caller observes. This is the level of explanation expected in a senior code review or technical interview: not just which pattern was selected, but which invariant it protects and what evidence shows that recovery works.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high-cardinality telemetry. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after evidence identifies a bottleneck or risk.

When the topic involves an external dependency, define a timeout, cancellation strategy, retry budget, and behavior after the budget is exhausted. When it involves persistence, define transaction boundaries and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, pending, and success states. When it involves security, assume the client can be modified and all network input is untrusted.

For distributed workflows, also inspect operational signals: outbox age, queue depth, consumer lag, retry counts, compensation failures, duplicate rates, and the number of workflows requiring manual review. These measurements turn “the saga is stuck” into a diagnosable boundary rather than a vague incident.

Guided lab

Design checkout across order, payment, inventory, and shipping. Use local transactions plus an outbox and either saga orchestration or choreography. Document every retry, compensation, and irrecoverable side effect. Your design should say where workflow state lives, how messages are correlated, how duplicate delivery is handled, and what status the client sees while the workflow is still progressing.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

Edge cases and failure modes

Local transactions: test absent and malformed input, uniqueness conflicts, rollback behavior, duplicate requests, and ordering or concurrency where applicable. Test the smallest and largest credible sizes, including transaction time and lock behavior.
Saga: test absent and malformed input, a failure at every step, delayed responses, retries after timeouts, duplicate commands or events, ordering or concurrency where applicable, compensation failure, and a workflow that reaches manual review. Verify that recovery is safe to repeat at the smallest and largest credible workflow sizes.
Orchestration and choreography: test absent and malformed input, a coordinator restart, a missing subscriber, event cycles, duplicate delivery, out-of-order delivery, ordering or concurrency where applicable, and the ability to reconstruct workflow state from logs and durable records. Check the smallest and largest credible workflow sizes.
Transactional outbox: test absent and malformed input, a crash after the business commit, a crash after publication, publisher retries, duplicate publication, ordering or concurrency where applicable, outbox growth, and a record that cannot be processed. Verify that consumers are idempotent at the smallest and largest credible event sizes.
Change data capture: test absent and malformed input, schema changes, deletes, snapshots and replays, lag, duplicate changes, ordering or concurrency where applicable, and a consumer that falls behind. Confirm that row-level changes still carry the semantics the consumer expects at the smallest and largest credible data sizes.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Treating a saga as a distributed database rollback, or assuming compensation always restores the original state.
Assuming an outbox or CDC pipeline provides exactly-once behavior without idempotent consumers and replay rules.
Testing only the happy path and discovering the real contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case and inspect the actual state transition, message ID, correlation ID, retry count, and timestamps. Trace the boundary where the invariant first became false: source or build, browser or DOM, Network or HTTP, server or route, database or query, or deployment or configuration. Compare the expected event and state sequence with the observed sequence. Then fix the owning layer instead of adding a downstream patch that hides the original failure.

Interview questions

What problem do local transactions solve, and what trade-off or failure mode would make you choose a different approach?
What problem does a saga solve, and how would you design compensation when a later step fails?
What problem do orchestration and choreography solve, and when would explicit workflow ownership be worth the extra component?
What problem does a transactional outbox solve, and what must consumers do when publication is at-least-once?
What problem does change data capture solve, and when would a domain event be a better contract than a row change?

Checkpoint

Without notes, explain Distributed Transactions, Sagas, Transactional Outbox, CDC, and Compensation to another developer in five minutes. Include one invariant, one edge case, one production failure mode, and one alternative design. Explain which service owns the invariant and what evidence would let you diagnose a stuck or duplicated workflow. Then implement a small example without copying the lesson code.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.