265: Rate Limiting, Quotas, Backpressure, and Admission Control

Learning outcomes

By the end of this lesson, you can:

explain and apply token bucket in a realistic implementation;
explain and apply leaky bucket in a realistic implementation;
explain and apply fixed/sliding windows in a realistic implementation;
explain and apply distributed limits in a realistic implementation;
explain and apply quotas and dimensions in a realistic implementation.

These are not interchangeable labels for “reject some requests.” You should be able to connect each mechanism to a requirement, describe the invariant it provides, and explain what happens when the limiter or one of its dependencies is under pressure.

Prerequisites and retrieval

This lesson assumes the earlier 01–06 foundation and the preceding lessons in this module. Before reading, retrieve one concrete example from a previous project where the same concern appeared. Perhaps an endpoint needed protection from bursts, a background worker could not keep up with a queue, or a shared API imposed a per-customer allowance. The example does not need to have used these names.

The point of the retrieval exercise is to connect the vocabulary to a real design decision. In a large-scale distributed service, the requirements, traffic shape, failure modes, cost, and operational constraints all need to be explicit. “Ten requests per second” is not enough by itself: you also need to know whether short bursts are acceptable, which identity is being limited, whether a limit is global, and what the caller should see when capacity is unavailable.

Terminology

Token bucket: Tokens accumulate up to a burst capacity, and requests consume tokens. This allows a bounded burst while preserving a sustained refill rate. The bucket therefore expresses two different limits: how much work can happen immediately and how quickly capacity returns.
Leaky bucket: A leaky-bucket/queue model smooths output at a roughly fixed rate and can bound bursts differently from token buckets. Incoming work may wait in a bounded queue, or it may be rejected when that queue is full.
Fixed/sliding windows: Fixed counters are simple but allow boundary bursts; sliding log/window approximations provide smoother enforcement with extra storage and complexity. The choice determines how much enforcement error and state the system accepts.
Distributed limits: Multiple gateways need shared or partitioned counters, approximate local quotas, or consistent hashing. A strong global limit adds coordination latency and creates availability trade-offs when the shared state is slow or unavailable.
Quotas and dimensions: A service can limit by user, tenant, IP, API key, endpoint, cost units, or concurrent jobs according to abuse and fairness requirements. A quota is usually about consumption over a longer period; a rate limit is about the pace of consumption, although a system may enforce both.
Backpressure: Consumers should signal or reflect saturation through bounded queues, concurrency limits, 429/503 responses, or producer slowdown instead of accepting unbounded work into memory. Backpressure is a capacity-protection mechanism, not merely a nicer error response.
Admission control: Admission control decides whether work should enter a system at all. It uses signals such as the selected rate limit, available concurrency, queue depth, priority, and dependency health to keep accepted work within a safe operating envelope.

Mental model

Treat Rate Limiting, Quotas, Backpressure, and Admission Control as a design problem with observable inputs, outputs, invariants, and failure modes. A request arrives with dimensions such as identity, endpoint, operation cost, and priority. The system evaluates those dimensions against available capacity, then admits, delays, or rejects the work. The result should be predictable enough that a client can retry safely or change its behavior.

Systems should reject excess work predictably before saturation causes universal timeouts. Rate limiting primarily protects fairness and security; backpressure primarily protects downstream capacity; admission control is the decision point that combines those concerns. They overlap, but they are not the same thing. A user can be within a request-rate limit while a database is overloaded, and a healthy queue can still allow one tenant to consume an unfair share unless quotas are dimensioned correctly.

A strong implementation makes assumptions visible, narrows uncertainty at boundaries, and leaves enough evidence—tests, types, constraints, metrics, or diagrams—to prove why the design is safe. Useful observations include allowed and rejected counts, queue depth, wait time, limiter-store latency, remaining quota, and the reason for rejection. Without those signals, a 429 or 503 tells the caller what happened but leaves the operator guessing why.

A useful interview and production sequence is:

requirement -> constraints -> model -> implementation -> failure analysis -> verification

Do not jump from a requirement directly to a library call. First state what must remain true. For example, “a tenant may submit at most 100 cost units per minute, with no more than 10 jobs running at once” is a more useful starting point than “use Redis rate limiting.” Then choose the mechanism that enforces that contract, decide where state lives, and define what happens when the mechanism itself fails.

Deep dive

1. Token bucket

When an API must tolerate a short burst but still limit its long-term load, a token bucket is often a natural model. Tokens accumulate at a configured refill rate up to a burst capacity, and each request consumes one or more tokens. A request can proceed immediately if enough tokens exist; otherwise it is rejected or, in a design that explicitly supports waiting, delayed.

The burst capacity and refill rate must be chosen separately. A bucket with a refill rate of 10 tokens per second and a capacity of 100 does not mean “exactly 10 requests every second.” It permits up to 100 available requests at once, followed by sustained use at roughly 10 requests per second. If operations have different costs, consume a cost-based number of tokens rather than pretending that an expensive export and a cheap lookup are equivalent.

There is a subtle implementation detail around time. Refill calculations need a monotonic notion of elapsed time where possible, and the state update must be atomic when multiple requests can inspect the same bucket concurrently. Otherwise two requests can both observe the same balance and overspend it. The implementation also needs defined behavior for clock precision, negative elapsed time, and a bucket that has been idle long enough to refill completely.

Decision rule: Use token bucket deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Ask whether bounded bursts are actually desirable, whether waiting is allowed, and whether the cost of each request is uniform.

2. Leaky bucket

The useful distinction from a token bucket is what the system smooths. A leaky-bucket/queue model lets work enter a bounded queue and drains it at a roughly fixed rate. The queue absorbs a limited burst, while the drain rate controls the output seen by a downstream service. If the queue is full, the producer must slow down or the request must be rejected.

This model is useful when downstream work needs a steadier pace, such as sending requests to a fragile third-party API or processing jobs through a worker pool. It can introduce latency because accepted work may wait. That latency is part of the contract, not an accidental side effect. A queue with no maximum length has simply moved the overload problem into memory, so the queue bound and the behavior at capacity need to be explicit.

Token and leaky buckets are related but make different promises. A token bucket commonly controls whether work may start now, while a leaky bucket commonly controls how quickly accepted work leaves the queue. A design may use both, but adding both without a reason makes capacity and failure analysis harder.

Decision rule: Use leaky bucket deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Specify the queue bound, maximum wait, ordering policy, cancellation behavior, and response when the queue is full.

3. Fixed/sliding windows

Fixed windows count requests during discrete intervals such as a minute. They are straightforward to store and explain, but their boundaries can create a burst: a caller may use its full allowance at the end of one window and its full allowance again at the beginning of the next. That behavior may be acceptable for a low-risk endpoint, but it should not be mistaken for smooth enforcement.

Sliding logs record individual request times and count the events in the most recent interval. This gives a more precise view of recent activity, at the cost of storing and removing more state. Sliding-window counters or weighted approximations reduce that storage cost but introduce approximation error. The decision is therefore about the acceptable boundary burst, memory use, state-management complexity, and accuracy under concurrency.

The key operational questions are what timestamp is trusted, how expired state is removed, and whether counter updates are atomic. A limit can look correct in a single-threaded test and still fail when several gateways increment the same key concurrently. Retry behavior also matters: a client that receives a rejection should have enough information, such as a retry hint when appropriate, to avoid turning the boundary into another synchronized burst.

Fixed counters are simple but allow boundary bursts; sliding log/window approximations provide smoother enforcement with extra storage/complexity.

Decision rule: Use fixed/sliding windows deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Choose the simplest window whose boundary behavior, precision, and storage cost meet the actual requirement.

4. Distributed limits

With one gateway, an in-memory counter may be enough for a local experiment. With multiple gateways, each instance sees only part of the traffic, so independent local counters can exceed the intended global limit. The system then needs shared state, partitioned ownership, approximate local quotas, or consistent hashing that routes a limiting key to a particular owner.

Shared counters can provide a stronger global limit, but every request may now pay coordination latency and depend on the availability of the limiter store. Partitioning reduces coordination but requires careful handling of hot keys and rebalancing. Approximate local quotas improve availability and latency, but the aggregate allowance can temporarily exceed the stated limit. None of these is universally correct; the choice depends on whether fairness, strict protection, availability, or low latency is the dominant requirement.

Failure behavior must be part of the design. If the central store is unavailable, fail-open behavior protects request availability but can expose the dependency to overload or abuse. Fail-closed behavior protects the dependency but may reject legitimate traffic. A bounded local fallback, a short cached decision, or a reduced emergency allowance can be safer than an unqualified choice, provided the bounds and recovery behavior are documented and tested.

Multiple gateways need shared or partitioned counters, approximate local quotas, or consistent hashing. Strong global limits add coordination latency and availability trade-offs.

Decision rule: Use distributed limits deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. State which limit is global, where its authoritative state lives, how hot keys behave, and whether a limiter outage is fail-open, fail-closed, or handled by a bounded fallback.

5. Quotas and dimensions

One global requests-per-second value rarely captures fairness. A service may need a per-user limit to stop one account from dominating, a per-tenant quota for billing or isolation, an IP limit for unauthenticated traffic, and an API-key limit for a particular integration. Endpoint and operation cost matter too: a search request, bulk export, and write may consume very different amounts of capacity.

Dimensions should come from the abuse and fairness model, not from every field available in the request. Each additional dimension adds state, cardinality, configuration, and debugging work. Identity resolution must happen at a trusted boundary; a client-supplied header is not automatically a trustworthy tenant identifier. For unauthenticated traffic, IP-based controls have limitations because many users can share an address and one user can change addresses.

Quotas also have a time horizon. A monthly tenant quota, a per-minute burst limit, and a maximum number of concurrent jobs protect different resources. Concurrency limits are especially important for work whose duration varies: ten long-running jobs can consume more capacity than hundreds of short requests. When several rules apply, define whether the most restrictive result wins and how the caller learns which dimension was exhausted.

Limit by user, tenant, IP, API key, endpoint, cost units, or concurrent jobs according to abuse and fairness requirements. One global requests/sec number may be insufficient.

Decision rule: Use quotas and dimensions deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Name the resource being protected, the identity used for accounting, the time period, the cost model, and the expected behavior when dimensions disagree.

6. Backpressure

Backpressure begins when a consumer cannot safely accept work at the producer's current pace. The consumer can reduce concurrency, bound a queue, slow the producer, or reject new work with 429 or 503. The right response depends on whether the overload is attributable to a caller's rate or to temporary service capacity, but in both cases the goal is to avoid turning overload into unbounded memory growth and cascading timeouts.

A bounded queue is meaningful only if the system has a policy for reaching its bound. It may reject newest work, preserve higher-priority work, shed duplicate work, or apply cancellation and deadlines. A worker concurrency limit should be paired with visibility into queue wait time and execution time; otherwise a request may appear admitted while spending most of its lifetime waiting. Producers and clients also need retry behavior that includes backoff and jitter where retries are safe. Blind retries can amplify the original overload.

Backpressure is different from a rate limit. A caller can respect its configured rate while a dependency slows down, and a fast producer can overwhelm a small worker pool even when no user-specific quota is exceeded. Admission control connects these signals: it should admit only work that the system can reasonably complete within its resource and reliability constraints.

Consumers should signal or reflect saturation via bounded queues, concurrency limits, 429/503, or producer slowdown instead of accepting unbounded work into memory.

Decision rule: Use backpressure deliberately when it makes the contract or invariant easier to prove. If it only reduces typing while hiding an assumption, prefer the more explicit design. Define the queue and concurrency bounds, the overload status, cancellation and timeout behavior, retry guidance, and the metric that tells operators whether the system is saturated.

Worked example

Consider a large-scale distributed service whose requirements, traffic, failure modes, cost, and operational constraints must be made explicit. Start by writing the requirement in one sentence, list the input and output contracts, and identify which of the concepts above owns each failure mode. For example, a tenant may have a sustained API allowance, a smaller burst allowance, and a limit on concurrently running jobs. Those are related controls, but they protect different resources and should not be collapsed into one unexplained number.

The important move is separation: parsing or validation belongs at the boundary; domain rules belong in the domain/service layer; persistence rules belong in the database or repository; presentation rules belong in the client. The limiter or admission decision usually sits at the boundary where identity and operation cost are known, while worker backpressure belongs near the queue and consumer. Mixing these concerns makes a happy-path demo look shorter but makes edge cases much harder to reason about. It also makes it unclear whether a rejected request was malformed, unauthorized, over quota, or refused because the service was saturated.

Client
  |
DNS -> CDN / Edge
  |
Load Balancer -> API instances -> Cache
                          |          |
                          +------> Primary datastore
                          |
                          +------> Queue / Stream -> Workers

The diagram gives each layer a useful question. The edge may apply coarse protection before traffic reaches the application. API instances can evaluate authenticated dimensions and decide whether work is admitted. The cache and primary datastore have their own capacity limits, while the queue and workers need bounded buffering and concurrency control. A request being accepted by the API does not prove that every downstream operation will succeed.

Walk the example with at least four cases: the normal path, an empty or missing value, a duplicate/retry/concurrent path where relevant, and a dependency failure. For each case, state which layer detects the problem and what the caller observes. For the normal path, identify the limit key, the token or counter change, and whether the work is immediate or queued. For an empty identity or malformed cost, reject at the boundary rather than accounting against an ambiguous key. For duplicate or concurrent submissions, decide whether the operation is idempotent and whether the limiter update and admission decision are atomic enough for the stated invariant. For a limiter-store or queue failure, state whether the service fails open, fails closed, or uses a bounded fallback.

This is the level of explanation expected in a senior code review or technical interview: not only which algorithm was selected, but what it guarantees, what it costs, and what the caller and operator observe when the assumptions stop holding.

Production perspective

Production correctness is broader than “the code works on my machine.” Ask how the design behaves during deploys, retries, partial failure, stale clients, concurrent requests, malformed data, schema changes, and high cardinality. A deploy can temporarily route traffic unevenly across gateway instances. A retry can consume capacity twice unless the operation and accounting model allow for it. A high-cardinality dimension can make the limiter's state store the new bottleneck.

Prefer explicit contracts, bounded resource usage, structured errors, and measurable behavior. Optimize only after you can identify the bottleneck or risk with evidence. A limit that is technically enforced but impossible to tune, inspect, or explain is still an operational problem. Record the configured allowance, rejection reason, queue delay, and dependency outcome without exposing sensitive identity data in logs.

When the topic involves an external dependency, define a timeout and cancellation strategy. When it involves persistence, define transaction and consistency expectations. When it involves user-visible state, define loading, empty, error, stale, and success states. When it involves security, assume the client can be modified and the network input is untrusted. In particular, do not treat a client-provided quota dimension or retry header as authoritative without validating how it was established.

Guided lab

Design per-user and per-tenant API rate limits plus a worker concurrency limit. Compare token bucket and sliding window, then describe behavior when the central limiter store is unavailable. Your design should say which dimensions are authoritative, whether a request costs one unit or a variable number of cost units, how bursts are handled, and whether accepted asynchronous work is placed in a bounded queue.

Complete the lab with this discipline:

Write the requirement and two non-requirements. For example, state whether a strict globally exact limit is required and whether the API is allowed to queue requests rather than reject them.
List input, output, and error contracts before implementation. Include the identity, operation cost, limit result, retry information, and the distinction between client throttling and service saturation.
Implement the smallest correct vertical slice. Keep the limiter decision, queue admission, and worker concurrency behavior observable rather than hiding them behind an unexplained abstraction.
Add at least one invalid-input test and one edge-case test. Useful cases include a missing identity, a full bucket, a window boundary, simultaneous updates, or a full worker queue.
Instrument or inspect the behavior instead of guessing. Look at allowed and rejected counts, remaining capacity, store latency, queue depth, and wait time.
Refactor one hidden assumption into an explicit type, constraint, function, or configuration. Examples include a named cost unit, a bounded queue size, or an explicit fail-open policy.
Explain one alternative design and why you did not choose it. Compare its accuracy, latency, state cost, availability, and operational behavior rather than calling it simply “less scalable.”
Record a short “what would break at 10× scale?” note. Consider hot keys, counter contention, cardinality, queue memory, worker saturation, and the effect of retries.

Edge cases and failure modes

Token bucket: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test refill after idle time, a request whose cost exceeds the bucket capacity, time precision, and two simultaneous consumers of the final available tokens.
Leaky bucket: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test a full queue, cancellation while waiting, drain-rate changes, maximum wait time, and whether rejected work is removed without being processed later.
Fixed/sliding windows: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test the fixed-window boundary burst, expired state, clock behavior, atomic increments, and approximation error in a sliding window.
Distributed limits: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test store timeouts, partitioned ownership, hot keys, gateway imbalance, fail-open versus fail-closed behavior, and recovery after the store becomes healthy.
Quotas and dimensions: test absence, malformed input, duplicates, ordering/concurrency where applicable, and behavior at the smallest and largest credible sizes. Also test shared IPs, changing identities, unknown tenants, competing dimensions, variable operation costs, quota reset boundaries, and concurrent jobs that run for different durations.

An edge case is not only an unusual input. It is any point where the invariant can become ambiguous: time crosses a boundary, two instances update the same state, a dependency stops responding, or accepted work lasts longer than expected. Those are the cases most likely to produce an outage even when the ordinary request path is correct.

Common mistakes and debugging

Solving the example instead of the requirement: a copied pattern can be syntactically correct but architecturally wrong. A token bucket copied from another endpoint may permit bursts that a downstream dependency cannot absorb.
Hiding uncertainty with assertions, broad exception handlers, permissive schemas, or “temporary” any values. A missing identity or unknown cost should not silently become a shared limiter key.
Testing only the happy path and therefore discovering contracts only after integration. Boundary timing, concurrent updates, full queues, and unavailable stores need deliberate tests.
Optimizing before measuring, or selecting a scalable mechanism without a scale requirement. A distributed counter may add more latency and failure surface than a local limit needs.
Letting client-side behavior stand in for server-side authorization, validation, or persistence guarantees. A client can omit a limiter call, alter an identity field, or retry regardless of the UI's intentions.

For debugging, reproduce the smallest failing case, inspect the actual value or execution plan, trace the boundary where the invariant first becomes false, and fix the owning layer rather than adding a downstream patch. Start by identifying the request's effective dimensions and the decision it received. Then compare the configured limit with the stored counter or bucket state, check timestamps and concurrency, inspect limiter-store latency and errors, and follow the request into the queue and worker metrics. A rising queue with normal request rates points to consumer capacity; a sudden increase in rejected requests at one gateway may point to partitioning or local state rather than a global traffic change.

Interview questions

What problem does Token bucket solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Leaky bucket solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Fixed/sliding windows solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Distributed limits solve, and what trade-off or failure mode would make you choose a different approach?
What problem does Quotas and dimensions solve, and what trade-off or failure mode would make you choose a different approach?

Answer each question with a requirement, not only an algorithm name. Include the protected resource, burst and time behavior, state location, expected status when capacity is exhausted, and what happens when the limiter dependency fails. A strong answer can also explain why backpressure or admission control is needed even after request-rate limiting is in place.

Checkpoint

Without notes, explain Rate Limiting, Quotas, Backpressure, and Admission Control to another developer in five minutes. Your explanation must include one invariant, one edge case, one production failure mode, and one alternative design. Then implement a small example without copying the lesson code.

The invariant might say that a tenant cannot consume more than its configured cost units in a period, or that the worker queue never exceeds its bounded capacity. The edge case should force you to discuss time boundaries, concurrency, or a full queue rather than a merely invalid string. The production failure mode should include the caller-visible behavior and the operator signal. Your alternative should make a real trade-off, such as exact shared coordination versus a bounded local approximation.

Mastery checklist

I can define the core terms precisely.
I can choose a design from requirements instead of from habit.
I can implement and test the normal path and edge cases.
I can explain the runtime, storage, or complexity cost.
I can identify which layer owns validation, errors, and recovery.
I can compare at least two reasonable alternatives.
I can explain how the design changes at larger scale or stricter reliability.