281: Case Study: Notification Platform

Learning outcomes

By the end of this lesson, you should be able to:

explain and apply event intake in a realistic implementation;
explain and apply preference evaluation in a realistic implementation;
explain and apply channel workers in a realistic implementation;
explain and apply scheduling and priority in a realistic implementation;
explain and apply deduplication in a realistic implementation.

Prerequisites and retrieval

This lesson assumes that you have the 01–06 foundation and the earlier lessons in this module. Before you start, retrieve one concrete example from a previous project in which one of these concerns appeared. Perhaps a service accepted an event and processed it later, honored a user's preferences, retried a provider call, or had to prevent the same action from happening twice.

The point is not to memorize a list of architecture terms. The point is to make a defensible decision inside a large-scale distributed service. That decision has to account for requirements, traffic, failure modes, cost, and operational constraints, and those assumptions need to be stated rather than left implicit.

Terminology

Event intake: Producers submit a stable notification intent or event containing the recipient, type, priority, template/version, and idempotency identity. They do not call every channel provider directly.
Preference evaluation: Centralized preferences, quiet hours, locale, and opt-outs determine which channels are eligible before dispatch. Security and transactional notifications may follow separate rules.
Channel workers: Email, SMS, push, and in-app delivery each have different provider APIs, rate limits, payload constraints, receipt semantics, and retryability. Workers isolate those differences from the rest of the system.
Scheduling and priority: Delayed notifications need a durable scheduler or queue. High-priority security messages should not sit behind millions of low-priority marketing jobs.
Deduplication: Retries and repeated upstream events must not turn into repeated messages that spam a user.
Delivery status: Track states such as accepted, queued, sent, delivered, and failed only to the level of accuracy the provider actually supports. Expose aggregate metrics, dead-letter queues (DLQs), and replay or administrative tools so operators can understand and recover from failures.

Mental model

Treat Case Study: Notification Platform as a design problem with observable inputs, outputs, invariants, and failure modes. A notification system is not just a loop that sends messages. It separates event intake, user preferences, channel routing, provider delivery, retries, deduplication, scheduling, templates, and observability. A strong implementation makes its assumptions visible, reduces uncertainty at each boundary, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe.

A useful sequence for both an interview and a production design is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement straight to a library call. First state what must remain true. Then choose the mechanism that enforces that invariant, and finally decide how you will verify it when a dependency is slow, a request is repeated, or traffic is much larger than the happy path suggests.

Deep dive

1. Event intake

The first design question is how a producer asks for a notification. Producers should submit a stable notification intent or event containing the recipient, type, priority, template/version, and idempotency identity. They should not need to know how to call every email, SMS, push, or in-app provider.

This boundary gives the platform a durable contract. It also gives the platform a place to validate the request, record the event, and apply later decisions without coupling every producer to provider-specific behavior. The intent is not the same thing as a successful delivery; it is the request that the notification workflow should begin.

Decision rule: Use event intake deliberately when it makes the contract or invariant easier to prove. If it merely reduces typing while hiding an assumption—for example, who owns retries or what idempotency means—prefer the more explicit design.

2. Preference evaluation

After intake, the platform needs to decide whether a user may receive the notification and through which channels. Centralized preferences, quiet hours, locale, and opt-outs determine the eligible channels before dispatch. Security and transactional notifications may have separate rules, so a general marketing preference should not silently suppress a message that the product treats as essential.

Keeping this decision in one place prevents each channel worker from interpreting preferences differently. It also makes the decision inspectable: when a message is not sent, operators should be able to distinguish an intentional opt-out from a malformed preference record or a dependency failure.

Decision rule: Use preference evaluation deliberately when it makes the contract or invariant easier to prove. If it merely reduces typing while hiding an assumption, prefer the more explicit design.

3. Channel workers

Email, SMS, push, and in-app delivery are not interchangeable transports. Each has its own provider API, rate limits, payload constraints, receipt semantics, and retryability. A channel worker owns those details and translates the platform's notification representation into the provider's request format.

That separation matters when a provider returns a timeout, a rate-limit response, or an acknowledgment whose meaning is narrower than “the user saw the message.” The worker can apply the channel's retry rules without making the intake or preference code understand every provider-specific failure.

Decision rule: Use channel workers deliberately when they make the contract or invariant easier to prove. If they only add indirection while hiding who owns provider errors or quotas, prefer the more explicit design.

4. Scheduling and priority

Some notifications are sent immediately; others are scheduled for a later time. Delayed work needs a durable scheduler or queue so that a process restart does not erase the schedule. Priority needs an equally explicit policy: a high-priority security message should not wait behind millions of marketing jobs.

Priority does not mean that every urgent request bypasses all controls. Provider quotas, user preferences, retry limits, and the distinction between accepted and delivered still apply. The design should make the ordering and isolation guarantees clear enough to test and observe.

Decision rule: Use scheduling and priority deliberately when they make the contract or invariant easier to prove. If they only make the architecture look more scalable while hiding the ordering, durability, or fairness assumptions, prefer the more explicit design.

5. Deduplication

Retries and repeated upstream events are normal in distributed systems. Without a deduplication rule, a timeout can lead to a retry while the original request is still being processed, and an upstream replay can send the same notification again. The result is user-visible spam.

Deduplicate by a stable notification or event key within a defined business window. The key and the window are product decisions, not guesses: two legitimate reminders may be different notifications, while two deliveries caused by the same event may be duplicates. Concurrency also matters, because two workers must not both conclude that they are the first one to process the same key.

Decision rule: Use deduplication deliberately when it makes the contract or invariant easier to prove. If it only hides an unclear event identity or retention policy, prefer the more explicit design.

6. Delivery status

A provider's acknowledgment does not necessarily mean that a user received or read a message. Track accepted, queued, sent, delivered, and failed states only to the accuracy the provider supports. Do not claim delivery certainty when the provider can report only acceptance.

The platform should also expose aggregate metrics, DLQs, and replay or administrative tools. Metrics show the shape of the problem; a DLQ preserves work that needs attention; replay tools support recovery after a transient dependency or configuration failure. These operational capabilities are part of the delivery design, not an afterthought.

Decision rule: Use delivery status deliberately when it makes the contract or invariant easier to prove. If it only creates a more detailed state model than the providers can support, prefer the more explicit design.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints all need to be explicit. Start with a one-sentence requirement. Then write down the input and output contracts and identify which of the concepts above owns each failure mode.

The useful distinction is separation of concerns: parsing and validation belong at the boundary; domain rules belong in the domain or service layer; persistence rules belong in the database or repository; presentation rules belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes edge cases and ownership much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Walk through at least four cases:

the normal path;
an empty or missing value;
a duplicate, retry, or concurrent path where relevant;
a dependency failure.

For each case, say which layer detects the problem and what the caller observes. For example, a boundary can reject malformed input, while a worker may need to classify a provider timeout for retry. That distinction is the level of explanation expected in a senior code review or technical interview: do not only name the component; explain the invariant, the failure owner, and the externally visible result.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high-cardinality workloads. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after evidence identifies a bottleneck or a meaningful risk.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and that network input is untrusted. In a notification platform, those assumptions affect both what the system accepts and what it is allowed to send.

Guided lab

Design multi-channel notifications with preferences, templates/locales, scheduled sends, provider quotas, priority lanes, retries/DLQ, idempotency, delivery receipts, and analytics.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

The lab is intentionally broad, but the implementation should still begin with a small vertical slice. For instance, establish one intake contract and one channel path before adding every provider and reporting dimension. The additional requirements then give you places to test ownership, ordering, quotas, retries, and operational recovery rather than encouraging a large unverified mock-up.

Edge cases and failure modes

Use the following as a minimum test matrix for each concept:

Event intake: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Preference evaluation: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Channel workers: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Scheduling and priority: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.
Deduplication: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes.

The repeated categories are useful because the same distributed-system hazards appear at different boundaries. The expected result is not necessarily “accept” or “retry” in every case; it is a documented, observable decision that belongs to the right layer.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and therefore discovering contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case first. Inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the layer that owns the problem rather than adding a downstream patch. For this case study, that may mean checking the accepted event before inspecting a worker, confirming the preference decision before blaming a provider, or examining queue and DLQ state before changing retry code.

Interview questions

What problem does Event intake solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Preference evaluation solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Channel workers solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Scheduling and priority solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Deduplication solve, and what trade-off or failure mode would make you choose a different approach?

Checkpoint

Without notes, explain Case Study: Notification Platform to another developer in five minutes. Include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

If your explanation names only components, it is incomplete. A good explanation also says what enters the system, what each boundary guarantees, what happens when work is repeated or delayed, and how an operator can tell whether the system is healthy.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.