Rate limiting

Budget caps bound total spend, but do nothing about burst rate — a tenant can fire an arbitrary number of steps per second until the cumulative cap trips. TokenBucketRateLimiter closes that gap: a classic token bucket, checked once per model step from the shared budget gate, so every architecture is paced through one seam.

from loomflow import Agent, TokenBucketRateLimiter
 
agent = Agent(
    "You are a support assistant.",
    model="gpt-4.1-mini",
    rate_limiter=TokenBucketRateLimiter(rps=5, burst=10),
)

That’s the whole wiring. Every model step now acquires one token from the calling user_id’s bucket before any provider call is made. The default is None — no limiter, and the call site is short-circuited entirely, so an agent without a limiter pays zero overhead.

TokenBucketRateLimiter(...)

TokenBucketRateLimiter(
    rps: float,                     # sustained refill rate (tokens/sec)
    burst: int,                     # bucket capacity
    *,
    per_user: bool = True,          # independent bucket per user_id
    mode: "throttle" | "raise" = "throttle",
    max_users: int | None = 100_000,
    user_idle_ttl_seconds: float | None = 86_400,
)

One tenant exhausting its bucket never affects another tenant’s — per-user isolation is the point.

Throttle vs raise

mode="throttle" queues: acquire sleeps until the refill rate frees a token, so bursts are smoothed rather than failed. Use it when the caller is a background job or an internal pipeline that should simply slow down.

mode="raise" fails fast with a typed error carrying a retry hint — for callers who’d rather surface “slow down” (e.g. propagate a 429 to their own clients) than queue work:

from loomflow import Agent, TokenBucketRateLimiter
from loomflow.core.errors import RateLimitExceeded
 
agent = Agent(
    "...", model="gpt-4.1-mini",
    rate_limiter=TokenBucketRateLimiter(rps=2, burst=4, mode="raise"),
)
 
try:
    result = await agent.run("hello", user_id="alice")
except RateLimitExceeded as exc:
    print(exc.user_id)       # "alice"
    print(exc.retry_after)   # e.g. 0.5 — seconds until the next token
    # → return HTTP 429 with a Retry-After header

RateLimitExceeded vs RateLimitError

Two deliberately distinct errors:

Catch RateLimitError for upstream quota problems; catch RateLimitExceeded for the framework’s own gate.

Budgets vs rate limits

The two govern different axes; production deployments usually want both.

from datetime import timedelta
from loomflow import Agent, TokenBucketRateLimiter
from loomflow.governance.budget import BudgetConfig, StandardBudget
 
agent = Agent(
    "...", model="gpt-4.1-mini",
    budget=StandardBudget(BudgetConfig(per_user_max_cost_usd=2.0)),
    rate_limiter=TokenBucketRateLimiter(rps=5, burst=10),
)

Custom limiters: the RateLimiter protocol

The in-process token bucket is one implementation of a structural protocol. To plug in a distributed limiter (Redis, an API gateway), implement one method:

from loomflow import Agent
 
class RedisRateLimiter:
    async def acquire(self, *, user_id: str | None) -> None:
        # take one permit from your shared store;
        # wait or raise RateLimitExceeded when none are available
        ...
 
agent = Agent("...", model="gpt-4.1-mini", rate_limiter=RedisRateLimiter())

The agent loop calls acquire once before every model step, passing the run’s user_id so per-tenant implementations can bucket independently.