Rate limiting

A rate limiter caps how fast each user can drive the agent, measured in model steps per second. Use one when many tenants share an agent and you need to smooth bursts or return HTTP 429s, rather than only cap total spend.

Budget caps bound total spend, but do nothing about burst rate: a tenant can fire any number of steps per second until the cumulative cap trips. TokenBucketRateLimiter closes that gap. It is a classic token bucket, checked once per model step from the shared budget gate, so every architecture is paced through one place.

from loomflow import Agent, TokenBucketRateLimiter
 
agent = Agent(
    "You are a support assistant.",
    model="gpt-4.1-mini",
    rate_limiter=TokenBucketRateLimiter(rps=5, burst=10),
)

That’s the whole wiring. Every model step now acquires one token from the calling user_id’s bucket before any provider call is made. The default is None: no limiter, and the check is skipped entirely, so an agent without a limiter pays zero overhead.

TokenBucketRateLimiter(...)

Every argument, with the keyword-only ones at their defaults:

from loomflow import TokenBucketRateLimiter
 
TokenBucketRateLimiter(
    5.0,                           # rps: sustained refill rate (tokens/sec)
    10,                            # burst: bucket capacity
    per_user=True,                 # independent bucket per user_id
    mode="throttle",               # or "raise"
    max_users=100_000,
    user_idle_ttl_seconds=86_400,
)

One tenant exhausting its bucket never affects another tenant’s; per-user isolation is the point.

Throttle vs raise

mode="throttle" queues: acquire sleeps until the refill rate frees a token, so bursts are smoothed rather than failed. Use it when the caller is a background job or an internal pipeline that should simply slow down.

mode="raise" fails fast with a typed error carrying a retry hint. Use it when you’d rather surface “slow down” (for example, return a 429 to your own clients) than queue work:

from loomflow import Agent, TokenBucketRateLimiter
from loomflow.core.errors import RateLimitExceeded
 
agent = Agent(
    "...", model="gpt-4.1-mini",
    rate_limiter=TokenBucketRateLimiter(rps=2, burst=4, mode="raise"),
)
 
try:
    result = await agent.run("hello", user_id="alice")
except RateLimitExceeded as exc:
    print(exc.user_id)       # "alice"
    print(exc.retry_after)   # e.g. 0.5: seconds until the next token
    # → return HTTP 429 with a Retry-After header

RateLimitExceeded vs RateLimitError

Two deliberately distinct errors:

Catch RateLimitError for upstream quota problems; catch RateLimitExceeded for the framework’s own gate.

Budgets vs rate limits

The two govern different axes; production deployments usually want both.

from datetime import timedelta
from loomflow import Agent, TokenBucketRateLimiter
from loomflow.governance.budget import BudgetConfig, StandardBudget
 
agent = Agent(
    "...", model="gpt-4.1-mini",
    budget=StandardBudget(BudgetConfig(per_user_max_cost_usd=2.0)),
    rate_limiter=TokenBucketRateLimiter(rps=5, burst=10),
)

Custom limiters: the RateLimiter protocol

The in-process token bucket is one implementation of a structural protocol. To plug in a distributed limiter (Redis, an API gateway), implement one method:

from loomflow import Agent
 
class RedisRateLimiter:
    async def acquire(self, *, user_id: str | None) -> None:
        # take one permit from your shared store;
        # wait or raise RateLimitExceeded when none are available
        ...
 
agent = Agent("...", model="gpt-4.1-mini", rate_limiter=RedisRateLimiter())

The agent loop calls acquire once before every model step, passing the run’s user_id so per-tenant implementations can bucket independently.