280: Case Study: Chat and Presence System

Learning outcomes

By the end of this lesson, you can:

explain and apply connection layer in a realistic implementation;
explain and apply message write path in a realistic implementation;
explain and apply ordering in a realistic implementation;
explain and apply fan-out in a realistic implementation;
explain and apply presence in a realistic implementation.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before you read, retrieve one concrete example from an earlier project where one of these concerns appeared. It might have been a notification stream, a background job, a collaborative editor, or a feature that synchronized state between devices. The domain does not need to be chat. What matters is that you can connect the terminology to a decision you have actually had to make.

The aim is not to memorize a list of distributed-systems terms. The aim is to make a defensible design decision inside a large-scale service. That decision has to account for requirements, traffic shape, failure modes, cost, and operational constraints. If those assumptions are left implicit, a design can sound plausible while failing as soon as a client retries, a node disappears, or a group becomes much larger than expected.

Terminology

The terms below describe separate responsibilities. They interact, but they are not interchangeable:

Connection layer: WebSocket/SSE gateways maintain client connections and authenticate sessions. They should be horizontally scalable, with shared routing and pub/sub, rather than treating in-memory connection state as the only copy of a durable message.
Message write path: The service persists a message with its conversation identity and an ordered ID or version before acknowledging it, subject to the stated durability requirements. It then publishes delivery events or otherwise makes the persisted message available to the delivery path.
Ordering: The system guarantees ordering within a conversation or partition rather than trying to define one global order for every conversation in the service. A global order is usually expensive and does not provide useful meaning across unrelated chats.
Fan-out: The service delivers messages to online devices through connection routing and retains enough unread or offline state for later synchronization. A delivery attempt is not the same thing as durable storage.
Presence: Heartbeats and lease expiry provide an approximate online or last-seen state. A sudden disconnect cannot be known everywhere instantly, so presence should not be presented as a perfect real-time fact.
Read receipts and sync: Receipts are state transitions that may arrive out of order when several devices act independently. Modeling a monotonic sequence or read position is generally safer than creating an unbounded boolean record for every message on every device.

The useful distinction is between a message being accepted, a message being stored, a message being delivered, and a message being read. Those events can happen at different times and can fail independently. A design that uses the word “sent” for all four states will be difficult to explain to users and difficult to debug.

Mental model

Treat Case Study: Chat and Presence System as a design problem with observable inputs, outputs, invariants, and failure modes. Chat combines persistent connections, message durability, per-conversation ordering, fan-out, offline delivery, approximate presence, and multi-device synchronization. None of these concerns can be proved by looking only at the happy-path request.

A strong implementation makes its assumptions visible, narrows uncertainty at each boundary, and leaves evidence that the design is safe. That evidence may be tests, types, database constraints, idempotency keys, metrics, logs, or diagrams. For example, “messages are ordered” is not yet an invariant you can verify. “Messages in one conversation have unique, increasing sequence numbers, and clients can request the next missing sequence” is much closer to one.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call. First state what must remain true. Then choose the mechanism that enforces it and identify what happens when that mechanism is unavailable. A WebSocket library can keep a socket open; it does not, by itself, provide durable storage, cross-node routing, ordering, or replay.

Deep dive

1. Connection layer

WebSocket and SSE gateways maintain client connections and authenticate sessions. Their job is to provide a live transport boundary, not to become the system of record for messages. A gateway may know that a device is connected to a particular process, but that knowledge is local and can disappear when the process restarts.

For horizontal scaling, a message intended for a user must be routable to the gateway holding that user's connection. Shared routing or pub/sub can distribute delivery events across gateway instances. The durable message remains in the message store so that a reconnecting or newly added device can synchronize from a known position instead of relying on an event that may already have been missed.

Authentication belongs at the connection boundary, but authorization still applies to every conversation and message operation. A valid session must not automatically grant access to every conversation. Also define what happens when a token expires while a socket remains open: the gateway may need to revalidate, close the connection, or require a refreshed credential.

Decision rule: Use the connection layer deliberately when it makes the transport contract or an invariant easier to prove. If it only reduces typing while hiding assumptions about authentication, routing, reconnection, or durability, prefer the more explicit design.

2. Message write path

The write path should persist a message with its conversation identity and ordered ID or version before acknowledging it, according to the durability promised to the caller. Only after the required write is complete should the system publish a delivery event, or it should use an outbox or equivalent mechanism so that a committed message cannot be lost merely because publishing failed immediately afterward.

This ordering creates an important boundary. A client may retry because the acknowledgement was lost even though the first write committed. The request therefore needs a stable client or server idempotency key, and the store needs a way to recognize a duplicate. Otherwise one user action can create two messages. The response should make clear whether it means “accepted for processing” or “durably stored.”

The write path also owns validation that affects persistence: conversation membership, message size, attachment references, and any required ordering or version checks. Do not rely on the client to enforce those rules. Client validation improves feedback, but the server is the authority because network input and client code are untrusted.

Decision rule: Use the message write path deliberately when it makes the durability, idempotency, and acknowledgement contract easier to prove. If it only reduces typing while hiding whether a message is committed or merely queued, prefer the more explicit design.

3. Ordering

Guarantee ordering per conversation or partition rather than globally. Participants in conversation A generally do not need messages in conversation B to share a position in the same global sequence. A per-conversation sequence is cheaper and gives clients a useful way to detect a gap.

Ordering is more than sorting by a timestamp. Clocks on different devices can disagree, and two writes can receive the same timestamp. Assign an authoritative sequence or version at the owning write boundary, then make retries and duplicate messages safe with stable client or server IDs and deduplication. Decide whether the contract is strict commit order, server acceptance order, or a looser display order; these are not the same promise.

A client can receive sequence 12 before sequence 11 because delivery is asynchronous. It should not silently treat 12 as proof that 11 does not exist. It can buffer briefly, request the missing range, or render a recoverable gap and synchronize. This is where ordering connects to offline sync: the durable sequence is the source used to repair an incomplete local view.

Decision rule: Use ordering deliberately when it makes the conversation-level contract and gap recovery easier to prove. If it only adds a counter without defining who assigns it, what retries do, or how clients recover gaps, the design is incomplete.

4. Fan-out

Fan-out delivers a persisted message to the online devices that should receive it through connection routing. It also preserves unread or offline state for later synchronization. A gateway being unavailable should delay live delivery, not erase the message or make the recipient permanently miss it.

One-to-one conversations and large group chats can require different strategies. Sending one independent event to every member is straightforward for a small group, but a very large group can turn one write into a costly burst. Fan-out-on-write, fan-out-on-read, batching, and hybrid approaches trade delivery latency, read cost, storage, and operational complexity differently. The right choice depends on group size, activity, and the required freshness.

The system should distinguish delivery to a device from delivery to a user. One user may have several phones, browsers, or desktop clients, each with its own connection and synchronization position. A device that reconnects should be able to ask for messages after its last confirmed position rather than depending on every live event having arrived.

Decision rule: Use fan-out deliberately when online routing, offline recovery, and the cost of recipient expansion are explicit. If it only means “broadcast this event” without a durable replay path or a plan for large groups, it is not a complete delivery design.

5. Presence

Heartbeats and lease expiry provide an approximate online or last-seen state. A connected client renews a lease periodically; if renewals stop, the service eventually marks the session stale or offline. The expiry window is a trade-off: a short lease reflects failures sooner but creates more heartbeat traffic and is more sensitive to temporary network delays.

Sudden disconnects are not instantly knowable everywhere. A process can crash before publishing an update, a network can partition, and a client can disappear without a clean close event. For that reason, presence should be treated as best-effort user-visible state, not as an authorization decision or a guarantee that a user can receive a message right now. Persisting a last-seen timestamp also requires a clear definition of which device or session it describes.

Decision rule: Use presence deliberately when the approximation, lease duration, update fan-out, and stale-state behavior are part of the contract. If it only exposes a boolean based on one process's memory, it will be misleading during failures and multi-device use.

6. Read receipts and sync

Receipts are state transitions that can arrive out of order across devices. A phone may report that a user read through message 20 after a desktop has reported message 15, while delayed network traffic later delivers the older report. The server should apply a monotonic rule so a stale receipt cannot move the conversation backward.

Where possible, store a read position or sequence rather than one boolean per message and device. A position such as “read through sequence 20” is compact, naturally idempotent, and lets the system derive which earlier messages are read. You may still need per-device positions when devices synchronize independently, but the model should be chosen intentionally rather than producing a record explosion by default.

Synchronization is the repair path for missed events. The client supplies its last known position, the server returns the missing durable range or an explicit indication that a full resync is required, and the client applies results idempotently. Define behavior for deleted messages, expired attachments, revoked access, and a conversation that changed while the sync request was in flight.

Decision rule: Use read receipts and sync deliberately when out-of-order updates, monotonic progress, and reconnect recovery are explicit. If it only toggles a boolean for the latest event, it will be vulnerable to stale devices and repeated requests.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start with one sentence such as: “A member can send a message to a conversation, and every authorized device can retrieve it in conversation order even if a gateway or client disconnects.” Then list the input, output, and error contracts. Decide which concept owns each failure mode instead of allowing every layer to make a slightly different decision.

The important move is separation. Parsing and basic validation belong at the boundary. Conversation membership, message rules, and idempotency belong in the domain or service layer. Persistence and sequence allocation belong in the database or repository. Delivery routing belongs in the messaging and connection infrastructure. Presentation and local optimistic state belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes retries, authorization failures, and partial completion much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                         |          |
                         +------> Primary datastore
                         |
                         +------> Queue / Stream -> Workers

The diagram is a conceptual flow, not a claim that every chat message should pass through a CDN or cache. The edge can terminate or route traffic, API instances can authenticate and coordinate writes, the primary datastore can hold the durable conversation record, and a queue or stream can decouple delivery work. A cache may help with appropriate reads, but it must not silently become the only place where an accepted message exists.

Walk the example through at least four cases:

Normal path: an authorized client submits a valid message, the write is committed with a conversation sequence, the acknowledgement is returned, and delivery workers notify online devices.
Empty or missing value: the request lacks conversation identity or message content, so the boundary rejects it with a structured client-visible error before persistence.
Duplicate, retry, or concurrent path: the client repeats a request after losing the acknowledgement, or two devices write concurrently. Stable IDs, membership checks, and the ordering rule determine whether the operation is deduplicated, serialized, or rejected.
Dependency failure: the datastore, stream, or gateway is unavailable. The service must say whether the message was not accepted, was durably stored but not yet delivered, or needs to be retried. It must not report successful delivery merely because a request reached an API instance.

For each case, state which layer detects the problem and what the caller observes. This is the level of explanation expected in a senior code review or technical interview. It also gives you a debugging map: if the acknowledgement says success but the message is absent after reconnect, inspect the write commit and replay position before debugging the socket.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failures, stale clients, concurrent requests, malformed data, schema changes, and high-cardinality traffic. A rolling deploy can move connections between gateway instances. A retry can repeat a committed write. A stale client can send an old read position. Each situation should have an intentional result rather than relying on timing.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Bound message and attachment metadata sizes, cap buffering for out-of-order events, and define backpressure when a recipient or downstream worker is slower than the incoming stream. Optimize only after you can identify the bottleneck or risk with evidence such as queue lag, write latency, connection counts, delivery delay, reconnect rate, or datastore contention.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and every network input is untrusted. In particular, attachment metadata should not be treated as proof that the caller may access the referenced object; authorization must be checked independently.

Guided lab

Design one-to-one plus group chat with multi-device users. Cover WebSocket gateways, message IDs and ordering, storage, online routing, offline synchronization, presence, receipts, attachment handling, and regional failure.

For attachment handling, keep the message path focused on authorized metadata and a durable reference rather than assuming that a large binary belongs in the same transaction as the message. Define size and type limits, access control, expiry or deletion behavior, and what the recipient sees if the attachment is unavailable. For regional failure, explain where a message is written, how failover affects ordering, and how a client reconciles a connection that reconnects to another region.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

The lab is intentionally open-ended. A good submission is not the one with the most infrastructure. It is the one that states its guarantees, makes failure observable, and explains which guarantees become weaker or more expensive at larger scale.

Edge cases and failure modes

Connection layer: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include expired credentials, reconnects, gateway restarts, and a device connected to more than one session.
Message write path: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include a lost acknowledgement after commit, a rejected unauthorized member, and a publish failure after durable storage.
Ordering: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include concurrent sends, delayed events, duplicate sequence notifications, and a client requesting a missing range.
Fan-out: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include offline devices, slow consumers, gateway loss, multiple devices per user, and a group whose membership changes during delivery.
Presence: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Include heartbeat loss, delayed heartbeats, process crashes, network partitions, and several sessions for one user.

Across all five areas, test not only whether an operation fails, but also what state remains afterward. A message may be durable while delivery is delayed; a presence value may be stale while the service is otherwise healthy; and a client may be connected while its synchronization position is behind.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong. A queue, cache, or WebSocket is not automatically justified by the word “chat.”
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values. These choices often turn a recoverable boundary error into corrupted state or an unexplained client timeout.
Testing only the happy path and therefore discovering contracts only after integration. Retries, stale devices, and partial failures should be test cases, not production surprises.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement. Fan-out and storage choices should follow group sizes, message rates, latency targets, and failure budgets.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees. An optimistic UI can make a product feel responsive, but it cannot prove that the server accepted or stored the message.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, and trace the boundary where the invariant first becomes false. For a missing message, compare the client request ID, the durable record, the assigned conversation sequence, the publish or queue event, and the device's last sync position. For incorrect presence, inspect lease renewals and expiry timestamps rather than trusting a single disconnect callback.

Fix the owning layer instead of adding a downstream patch. If duplicate messages originate because the write path has no idempotency key, filtering them in the UI only hides the data problem. If sequence gaps are caused by a broken replay query, increasing a gateway retry count does not restore correctness.

Interview questions

What problem does Connection layer solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Message write path solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Ordering solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Fan-out solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Presence solve, and what trade-off or failure mode would make you choose a different approach?

When answering, do not stop at naming a technology. State the invariant, the boundary that owns it, the failure mode that threatens it, and the operational cost of the mechanism you chose. For example, an answer about presence should mention approximation and lease expiry, not just heartbeats.

Checkpoint

Without notes, explain Case Study: Chat and Presence System to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

Your explanation should distinguish durable acceptance from live delivery, and it should say how a reconnecting client repairs missed events. If you cannot explain those boundaries, return to the write path, fan-out, and sync sections before treating the design as complete.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.