Token-Aware Rate Limiting: The Guardrail Request Limits Can't Provide

The abuse report where nobody broke a rule

The report was confusing because the user had not broken anything. They had not exceeded the rate limit, had not hammered the endpoint. The access logs showed a calm cadence, a request every couple of seconds, well under the 60-per-minute cap. Textbook good behavior.

They had also spent more in one morning than the next thousand users combined. Each polite, well-spaced request carried a 180,000-token document and asked for a long structured analysis of it. The rate limiter waved every one through, because it was counting the wrong thing. It was counting requests. The bill was denominated in tokens.

Requests are not the unit of cost

In a conventional API, requests are a fine proxy for load: each does roughly the same work, costs roughly the same to serve. Cap the request rate and you have capped the load. The whole industry built its rate-limiting muscle memory on this, and for conventional APIs it holds.

It does not hold for language models, because the work per request is unbounded. A single LLM request can carry a hundred tokens or the model's full context window, hundreds of thousands, and ask for a one-word answer or thousands of tokens of generation. The cost of a request can vary by three or four orders of magnitude while the request count stays at exactly one. Pricing reflects this directly: you are billed per million input and output tokens, not per call. So a request-per-minute limit is a speed limit measured in trips, on a road where one trip might be a bicycle and the next a fully loaded freight train.

This is why the providers limit on TPM

The providers learned this early, which is why their own limits are denominated in tokens per minute alongside requests per minute, and in practice TPM is the one you hit first because it tracks the resource actually being consumed (API rate limits compared across providers). The providers protect themselves with token-aware limits. Most teams then protect their users with request-aware limits, leaving a wide gap between what the provider meters and what the application controls. That gap is where the expensive user lives: under your RPM limit, well within the provider's account-wide TPM, spending your headroom on your dime, perfectly legally.

What a token-aware limiter looks like

Rate-limit on tokens, per user, with a token bucket. The bucket is the right primitive because it separates sustained rate from burst, which maps cleanly onto how token spend behaves: a baseline of normal usage plus occasional large legitimate requests you want to permit without permitting forever. The bucket has a capacity (burst allowed) and a refill rate (sustained throughput). Each request costs tokens equal to its size; if the bucket has enough, it proceeds and drains, otherwise it is throttled.

class TokenBudgetBucket:
    def __init__(self, capacity, refill_per_sec):
        self.capacity = capacity          # max burst, in tokens
        self.refill   = refill_per_sec    # sustained tokens/sec
        self.tokens   = capacity
        self.last     = time.monotonic()

    def allow(self, cost_tokens):
        now = time.monotonic()
        self.tokens = min(
            self.capacity,
            self.tokens + (now - self.last) * self.refill,
        )
        self.last = now
        if self.tokens >= cost_tokens:
            self.tokens -= cost_tokens
            return True
        return False

The subtlety unique to LLMs: you do not know the full cost until the call finishes, because output tokens stream out on the fly. So charge in two phases. Before the call, debit the known input tokens plus a conservative estimate of the output, capped by max_tokens. After the call, reconcile against the actual usage the provider returns, refunding or charging the difference.

Make the pre-charge pessimistic, because the risk is asymmetric: if you under-estimate and only reconcile afterward, a flood of requests can all pass on optimistic estimates and blow the budget once the tokens are already spent and unrecoverable, you cannot un-send a request. Assume close to the full max_tokens on the way in, then refund what came back small. Pessimistic in, honest out. And weight output by its real price, output commonly costs ~5x input on frontier models, so charge the bucket in cost-weighted units, or a user generating long outputs sails under a limit tuned for input-heavy traffic.

Per-user, per-tenant, and the noisy neighbor

A single global token limit protects your account from the provider's TPM ceiling but does nothing about fairness: one heavy user inside a shared global limit starves everyone else, the noisy-neighbor problem, now denominated in dollars. The answer is a hierarchy of buckets, per-user, per-tenant, global, each enforced independently, and a request must pass all three. The per-user bucket stops one account running away; the per-tenant bucket contains the blast radius to one customer; the global bucket protects against the provider limit and a correlated spike across many users.

Set the per-user refill rate from your unit economics, not a round number. If a subscription is worth $20 a month, size the sustained rate so that even flat-out a user cannot cost you more than that, otherwise you ship a plan that loses money the harder it is used. We make the broader version of this argument in Cost Per Conversation Under Load: every guardrail should trace back to the unit economics, or it is a number someone guessed.

Abuse, and the cost-amplification attack

Token-aware limiting is also a security control, because LLM endpoints have an attack class conventional APIs do not: cost amplification. An attacker does not need to take your service down, they need only make it expensive. A handful of requests, each maxing the context window and demanding maximum output, runs up real money while staying under any request-rate limit. OWASP tracks this as unbounded consumption in its Top 10 for LLM Applications: resource use with no ceiling becomes denial of wallet, not just denial of service (OWASP LLM10: Unbounded Consumption).

The satisfying part is that the defense is the same as the billing defense: a per-user token bucket sized to your economics stops the runaway customer and the malicious one alike. The attacker cannot exceed the budget any more than the heavy user can, because the limiter does not care about intent. It cares about tokens. Bound the tokens and you bound both the bill and the attack.

Reject loudly, degrade gracefully

When a user hits their token budget, return a structured 429 with a Retry-After and a body that explains it is a token budget, not a request-rate issue, so a legitimate heavy user understands and a client backs off correctly. For degradation, prefer routing over hard rejection: a user who has burned their premium-model budget can be transparently downshifted to a cheaper, smaller model rather than cut off, converting a hard wall into a soft slope, usually the right product call.

Test the giant request, not the fast one

Load tests for rate limiting almost always test the wrong attack. They fire thousands of small identical requests as fast as possible and confirm the limiter throttles them. It does, that proves the RPM limiter works, and proves nothing about the failure that empties your account. The test that matters is the opposite shape: a small number of enormous, well-spaced requests, each near the context-window ceiling with maximum output. Assert the token limiter throttles them even though the request rate is trivial. Assert it on the per-user, per-tenant, and global buckets independently. Assert the two-phase accounting reconciles when actual output differs from the estimate, both smaller (refund) and at the cap (full charge). Assert the cost-amplification attacker is bounded by exactly the budget you set.

assert limiter.allow(user="u1", input_tokens=180_000, max_output=4_000) is False, \
    "giant single request slipped past the token budget"
assert limiter.allow(user="u1", input_tokens=200, max_output=200) is True, \
    "small request wrongly throttled while budget remained"

Count the thing that costs money

Every rate limiter answers one question: how much of this will we allow? The only honest version for an LLM is denominated in tokens, because tokens are what you are billed for, what an attacker can amplify, and what one well-behaved user can consume a thousand times faster than the next. Request limits answer a different question, how many trips, and that stopped being a good proxy the moment one trip could be a freight train. Keep the request limit for your connection pool, but put the real guardrail where the real cost is. Count tokens, bucket them per user, weight them by price, and size the bucket to what a user is worth. Then the polite, account-emptying request hits a wall it cannot talk its way around.