Prompt caching

Prompt caching is the single biggest cost lever for an agent with a large stable system prompt. Typically 50-90% off the input bill, with a measurable latency drop too. Every lab implements it differently. You don’t have to care:

from loomflow import Agent
 
agent = Agent(
    LARGE_SYSTEM_PROMPT,
    model="claude-opus-4-7",
    prompt_caching=True,
)

prompt_caching=True does the right thing per provider.

Why it pays off

Your system prompt is the same on every turn. Domain instructions, tool catalog, few-shot examples, framework rules. None of it changes between user prompts. Without caching, you pay full input rate to re-send all of it every single turn.

Caching makes the provider remember that prefix. The second turn, the third turn, every turn after, the stable part is served from cache at a steep discount.

Per-provider behavior

For OpenAI, caching happens on the provider side whether or not you set the flag. What prompt_caching=True buys you is accurate cost accounting (loomflow reads cached_tokens and applies the discount in cost_usd) plus the optional cache_key routing hint.

For Anthropic, the flag is load-bearing. Without it, no cache_control markers get injected and you pay full rate.

DeepSeek and OpenAI-compatible gateways

DeepSeek’s own API caches automatically — like OpenAI, there is nothing to opt into, and ttl is meaningless on this path. The only wire-visible knob loomflow sends is cache_key → prompt_cache_key.

That knob matters far more than it looks once a failover gateway sits between you and the model. A gateway that spreads requests across upstream hosts sends consecutive turns of the same conversation to different backends — so the prefix cache never gets a second look and every turn bills at full input rate, even though “caching is automatic.” The cache key pins consecutive requests to one host, and hits start on the very next call:

agent = Agent(
    SYSTEM_PROMPT,
    model=OpenAIModel(
        "deepseek-v4-pro",
        api_key=GATEWAY_KEY,
        base_url="https://gateway.example.com/v1",
        # Gateways resell at their own rates; the 3-tuple's third
        # element is the ABSOLUTE cached-input rate in USD/MTok, so
        # cost_usd and budget caps stay honest.
        cost_per_mtoken=(0.75, 3.00, 0.008),
    ),
    prompt_caching={"enabled": True, "cache_key": run_id},
)

This is exactly the tool-loop workload caching was built for: an agent resends the same growing conversation on every pass, so from the second pass on, the repeated prefix bills at the cached rate. The accounting works on both wire shapes — loomflow reads prompt_tokens_details.cached_tokens and falls back to DeepSeek’s native prompt_cache_hit_tokens.

The dict form

For advanced control, pass a dict instead of True:

agent = Agent(
    LARGE_SYSTEM_PROMPT,
    model="claude-opus-4-7",
    prompt_caching={
        "enabled": True,
        "ttl": "1h",              # "5m" (default) or "1h" — Anthropic only
        "cache_key": "session_42", # OpenAI prompt_cache_key routing hint
    },
)

The three accepted shapes all resolve to a PromptCacheConfig: False / None disables it (the default), True enables it with a 5-minute TTL, a dict gives you per-field control.

Reading the numbers

RunResult carries the cache accounting:

result = await agent.run("...", user_id="alice")
 
result.tokens_in          # prompt tokens billed at full rate (cache misses)
result.cached_tokens_in   # prompt tokens served from cache
result.cache_write_tokens # tokens written to cache this run (Anthropic only)
result.tokens_out         # completion tokens
result.cost_usd           # already reflects the cache discounts

cost_usd is computed from all four token buckets against the model’s pricing entry, so the discount is already baked in. You don’t apply it yourself.

The same fields exist per-call on Usage as cached_input_tokens and cache_write_tokens. Total tokens the model actually processed is tokens_in + cached_tokens_in.

Watching it work

The cleanest proof is two back-to-back runs with the same big prompt. The first run is a cache miss. The second run shows non-zero cached_tokens_in and a markedly lower cost_usd.

agent = Agent(LARGE_SYSTEM_PROMPT, model="claude-opus-4-7", prompt_caching=True)
 
r1 = await agent.run("First question", user_id="alice", session_id="s1")
# r1.cache_write_tokens > 0  (cache cold, tokens written at 1.25x)
 
r2 = await agent.run("Second question", user_id="alice", session_id="s1")
# r2.cached_tokens_in > 0    (cache hit, served at 0.1x)
# r2.cost_usd noticeably lower than r1.cost_usd

The default cache window is 5 minutes, so the two runs need to happen back-to-back. Example 17 runs this exact comparison against both Anthropic and OpenAI.