Token-budgeted injection

At the start of every run the agent injects recalled memory (working blocks, facts, episodes) as a system block above the chat history. Tuning(memory_token_budget=) caps how many tokens that block may use, and can prefer fresh memories over old ones. Use it when memory items can be long or your model’s context window is small.

Without a budget, the block is limited by item count only (5 facts, 3 episodes, all working blocks), so one oversized episode can blow the context. The budget puts a hard token ceiling on the whole memory block, with optional recency decay:

from loomflow import Agent, Tuning
 
agent = Agent(
    "Answer using what you know about the user.",
    model="gpt-4.1-mini",
    memory="sqlite:./bot.db",
    tuning=Tuning(
        memory_token_budget=1500,          # cap the seed memory block
        memory_decay_half_life_days=30.0,  # optional: prefer fresh memories
    ),
)

What the budget covers

Three memory layers feed the seed, and the budget treats them differently:

1. Working blocks are pinned. They are injected first and count against the budget, but are never dropped or truncated, so a user-profile block always reaches the model in full. Oversized blocks shrink the allowance left for recall instead.
2. Facts + episodes fill the remainder. They are merged into one list, scored, and greedily packed into whatever the working blocks left over, then rendered as a single Recalled memory (most relevant first): block.
3. Session history is untouched. The budget governs cross-session recall only. This conversation’s rehydrated turns are separate (see Run vs session), and mid-run transcript growth is auto-compact’s job, not this budget’s.

Scoring: relevance × decay

Each candidate item is scored relevance * decay:

• Relevance is the item’s reciprocal rank within its own list, 1 / (60 + rank + 1) (the RRF term). Facts are ranked in the order the fact store recalls them; episodes in recall_scored() order when the backend supports it, plain recall order otherwise. The best fact and the best episode therefore score the same, and ties alternate fact, episode, fact. (Before 0.17 every fact got a fixed 1.0, which outranked every episode.)
• Decay is exponential: 0.5 ** (age_days / half_life_days). A 30-day half-life halves an item’s score every 30 days. Facts decay on recorded_at, episodes on occurred_at; items without a timestamp never decay. memory_decay_half_life_days=None (the default) means no decay, so ordering is by relevance alone. Neighbouring ranks differ by under 2%, so once a half-life is set, age does most of the ordering.

Packing is greedy, best-score-first. The first item that doesn’t fully fit is truncated to the remaining allowance with an …[truncated] marker, and packing stops there. The top-scored item is always included (truncated if even it doesn’t fit), so a tiny budget still surfaces the single most relevant memory rather than nothing.

# Fresh episode outranks a year-old one at equal relevance r:
# score(fresh, 1 day)   = r * 0.5 ** (1/30)   ≈ r * 0.977
# score(old, 365 days)  = r * 0.5 ** (365/30) ≈ r * 0.0002

Token costs use the framework’s chars/4 heuristic, the same estimator used for tool-result caps and tool-definition budgeting.

Without a budget

With memory_token_budget=None (the default), injection uses fixed item counts: up to 5 facts and 3 episodes under separate Known facts: / Relevant past episodes: headers, plus every working block. Setting a half-life without a budget changes nothing, because decay only applies under a budget.

Tuning()                                    # item-count limits
Tuning(memory_decay_half_life_days=30.0)    # still item-count limits: no budget set
Tuning(memory_token_budget=1500)            # budgeted, no decay

Choosing a budget

A useful mental model: the budget is a cap and a ranking policy, not a retrieval expansion. The limits stay the same (up to 5 facts and 3 episodes reach the seed); the budget decides how much of that actually ships, and in what order.

The 3 episodes are from other sessions. This session’s own turns are already in the chat history, so recall fetches 12 episodes, drops the current session’s, and keeps the top 3 of the rest. (Before 0.17 it fetched only 3, so a cross-session answer was lost whenever this session’s turns ranked first.)

For what working blocks, facts, and episodes are, see the Memory overview and Bi-temporal facts.