284: Case Study: File Storage and Sync

Learning outcomes

By the end of this lesson, you can:

explain and apply metadata versus content in a realistic implementation;
explain and apply chunking and deduplication in a realistic implementation;
explain and apply versioning in a realistic implementation;
explain and apply sync protocol in a realistic implementation;
explain and apply offline conflicts in a realistic implementation.

Prerequisites and retrieval

This lesson assumes that you have worked through the earlier 01–06 foundation and the preceding lessons in this module. Before you start, retrieve one concrete example from an earlier project where one of these concerns appeared. It might be a file upload, a cached resource, an optimistic update, or a background job that had to catch up after a client was offline.

The point is not to memorize a set of architecture terms. The point is to make a defensible decision inside a large-scale distributed service. That requires making the requirements, traffic, failure modes, cost, and operational constraints explicit before choosing an implementation.

Terminology

Metadata versus content: Keep folder and file identities, ownership, versions, and content pointers in a transactional metadata store. Store the actual content blocks in object storage, which is better suited to large, durable blobs.
Chunking and deduplication: Split a large file into chunks, hash those chunks, upload only the chunks that are missing, and build a file version from ordered chunk references. Encryption and privacy requirements can limit deduplication across users.
Versioning: Represent each update as a new version or optimistic revision so that a stale client cannot silently overwrite a newer update.
Sync protocol: Have clients maintain a cursor or change sequence, upload local changes, fetch remote deltas, and resolve conflicts when both sides changed from the same base version.
Offline conflicts: Text may be mergeable, while arbitrary binary files often require a conflicted copy or an explicit last-writer policy.
Garbage collection: A chunk must not be deleted while it is referenced by any version that the system still retains.

Mental model

Treat Case Study: File Storage and Sync as a design problem with observable inputs, outputs, invariants, and failure modes. A Dropbox-like service is not just an upload endpoint. It combines large-object transfer, metadata and versioning, block-level deduplication, conflict resolution, offline clients, notifications, and garbage collection.

A strong implementation makes its assumptions visible and narrows uncertainty at system boundaries. It also leaves enough evidence—tests, types, constraints, metrics, or diagrams—to show why the design is safe. A design that works only along the happy path is not yet a file-sync design; it is a demo of one successful request.

A useful sequence for both interviews and production work is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement straight to a library call. First state what must remain true. Then select the mechanism that enforces that invariant, and finally decide how you will observe and test it.

Deep dive

1. Metadata versus content

The useful distinction is between the description of a file and the bytes that make up the file. Store folder and file identities, ownership, versions, and pointers in a transactional metadata store, while the content blocks live in object storage. The metadata store can then enforce relationships and ownership rules without forcing the database to handle every large binary payload.

This split is useful only when it makes the contract easier to reason about. A metadata record should not claim that a version is available before the chunks it references are available. The pointer is part of the relationship between the two stores, so the design must define what happens if the metadata write succeeds but a content operation fails, or if a retry observes an operation halfway through.

Decision rule: Use metadata versus content deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

2. Chunking and deduplication

Uploading one enormous file as one indivisible request makes retries expensive and makes a transient failure affect the whole transfer. Chunking splits the file into manageable pieces. Hash each piece, upload only the chunks that are missing, and compose a file version from the ordered chunk references.

The order matters: two files can contain the same chunks in different orders and therefore represent different content. The hash is useful for identifying a chunk, but the system still needs to consider authorization, storage ownership, and the privacy implications of reusing content. Encryption or privacy boundaries can reduce or eliminate cross-user deduplication options.

Decision rule: Use chunking and deduplication deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

3. Versioning

An update should create a new version or optimistic revision. That gives the system a way to detect that a client is stale instead of allowing the stale client to silently replace newer content. It also gives sync, restore, and garbage collection a history of which content is still meaningful.

There is one subtle ordering requirement: a metadata commit should reference fully available chunks. Otherwise another client can observe a version that is visible in metadata but cannot be downloaded completely. The implementation therefore needs an explicit preparation step, a commit step, and a failure or cleanup path when preparation does not finish.

Decision rule: Use versioning deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

4. Sync protocol

Sync is a protocol, not merely a periodic request to download the newest file. Clients maintain a cursor or change sequence, upload local changes, fetch remote deltas, and resolve conflicts when both sides changed from the same base. The cursor lets a client ask for what it has not seen rather than repeatedly downloading the entire history.

The protocol must define what a cursor means, when it can be advanced, and how retries behave. A client should not advance past a change it has not successfully applied. Upload and download operations also need a clear relationship to the base version, because a successful network request does not by itself prove that the change is still current.

Decision rule: Use sync protocol deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

5. Offline conflicts

Offline work creates a conflict when two clients change the same logical file from the same base version. Text may support a meaningful merge, but arbitrary binary files often cannot be merged safely. In that case, the service may create a conflicted copy or apply a last-writer workflow.

The policy must be visible to users. Silently discarding one client's work is not conflict resolution; it is data loss hidden behind a successful-looking sync. The choice also belongs to the content type and product requirements, not just to the transport layer.

Decision rule: Use offline conflicts deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

6. Garbage collection

Deduplication means that a chunk may be shared by several file versions. A chunk referenced by any retained version cannot be deleted. Use reference tracking or mark-and-sweep with grace periods so that an in-progress upload, delayed metadata update, or restore does not race with cleanup.

Garbage collection is therefore part of correctness, not only a storage-cost optimization. A conservative delay can retain unused data longer, but deleting too early can make a supposedly retained version unreadable. The design should also make the mark, sweep, and retention rules observable enough to investigate a missing or unexpectedly retained chunk.

Decision rule: Use garbage collection deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be explicit. Start by writing the requirement in one sentence. Then list the input and output contracts, and identify which concept above owns each failure mode.

The important architectural move is separation of concerns. Parsing and validation belong at the boundary. Domain rules belong in the domain or service layer. Persistence rules belong in the database or repository. Presentation rules belong in the client. Mixing these concerns can make a happy-path demo look shorter, but it makes retries, stale state, and partial failure much harder to reason about.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

Walk through the design with at least four cases:

The normal path: a client prepares chunks, commits a version, and another client receives the resulting change.
An empty or missing value: for example, a missing file pointer, an empty chunk list, or a request that omits a required base version.
A duplicate, retry, or concurrent path where relevant: the same chunk upload is retried, or two clients attempt to commit from the same revision.
A dependency failure: object storage, the metadata store, the queue, or a notification path becomes unavailable.

For each case, state which layer detects the problem and what the caller observes. This level of ownership is what you want in a senior code review or technical interview. It prevents an API from reporting success merely because one part of a distributed operation completed.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high-cardinality workloads. Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after evidence identifies the bottleneck or risk.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume that the client can be modified and that network input is untrusted.

For this case study, production questions include how interrupted uploads are resumed, how a stale cursor is repaired, how authorization is checked before revealing metadata or content, and how cleanup avoids deleting a chunk that a restore or upload is about to reference. These are operational forms of the same invariants introduced above.

Guided lab

Design file sync across desktop and mobile with chunked upload, deduplication, versions, delta-sync cursors, offline conflicts, sharing permissions, deletion and restore, and chunk garbage collection.

Complete the lab with this discipline:

Write the requirement and two non-requirements.
List the input, output, and error contracts before implementation.
Implement the smallest correct vertical slice.
Add at least one invalid-input test and one edge-case test.
Instrument or inspect the behavior instead of guessing.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration.
Explain one alternative design and why you did not choose it.
Record a short “what would break at 10× scale?” note.

Your design should make it possible to answer where a version becomes visible, how a retry avoids duplicating work, what a client does with a rejected base revision, and why garbage collection is allowed to delete a particular chunk. The lab is open-ended, but those contracts and explanations are not optional details.

Edge cases and failure modes

Metadata versus content: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Include the case where metadata and content operations do not complete together.
Chunking and deduplication: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that chunk order is preserved and that privacy or encryption rules do not accidentally expose reusable content.
Versioning: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Verify that a stale revision cannot silently overwrite a newer one and that visible versions reference available chunks.
Sync protocol: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Check retries, cursor advancement, missing deltas, and dependency failure.
Offline conflicts: Test absence, malformed input, duplicates, ordering and concurrency where applicable, and behavior at the smallest and largest credible sizes. Check both mergeable text and non-mergeable binary content, including the user-visible conflicted-copy or last-writer behavior.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong for the actual consistency, cost, or failure constraints.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values.
Testing only the happy path and discovering the contracts only after integration.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees.

For debugging, reproduce the smallest failing case first. Inspect the actual value, stored metadata, request sequence, or execution plan. Trace the boundary where the invariant first becomes false, then fix the layer that owns that invariant rather than adding a downstream patch.

In a sync system, that may mean comparing the client's base revision with the committed revision, inspecting the cursor before and after applying a delta, checking whether every referenced chunk exists, or following a retry through the API, datastore, queue, and worker logs. A downstream “file missing” error is often only the last symptom of an earlier metadata or ordering problem.

Interview questions

What problem does Metadata versus content solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Chunking and deduplication solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Versioning solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Sync protocol solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Offline conflicts solve, and what trade-off or failure mode would make you choose a different approach?

Checkpoint

Without notes, explain Case Study: File Storage and Sync to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.