Skip to Content
RecipesProduction checklist

Production checklist

A pre-ship checklist for an agent that real users will reach. Work through it before the first production deploy, and again before releases that change the agent loop, memory, or security settings.

Reliability

  • Durable runtime: runtime=SqliteRuntime(...) or PostgresRuntime(...), with a fixed session_id per job, so crashes don’t lose work. Add tuning=Tuning(checkpoint=True) to resume long runs with agent.resume(). See Checkpoints. Journal values are JSON; rows pickled by releases before 0.17 are skipped (those steps run again) unless you pass serializer="pickle", which is safe only for a store no untrusted party can write.
  • Persistent memory: pass a URL, memory="sqlite:./bot.db" for single-instance, memory="postgres://..." / memory="redis://..." for multi-instance. Not the default "inmemory" which loses everything on exit.
  • Multi-tenancy: pass user_id= and session_id= to every agent.run. Memory partitions automatically; no app-side namespace plumbing.
  • Per-user budget caps: BudgetConfig(per_user_max_tokens=, per_user_max_cost_usd=) so one tenant can’t exhaust another’s quota. See Per-user budget caps.
  • Bounded in-process state: StandardBudget and InMemoryMemory default to 100k users + 24h idle TTL. For known smaller tenant pools, lower max_users to reclaim memory faster.
  • Auto fact extraction: on by default for real models; facts the user tells the bot persist as structured triples for future runs to recall. Pass auto_extract=False to opt out. Extraction runs in the background, so call await agent.aclose() at shutdown to let it finish.
  • Budget: StandardBudget with max_tokens, max_cost_usd, max_wall_clock. Soft warnings at 80%. The wall clock totals time spent running (idle time doesn’t count); cap a single run with Agent(timeout=).
  • Max turns cap: default 50; lower if your tools are expensive.
  • Rate limits: rate_limiter=TokenBucketRateLimiter(...) when many tenants share an agent. See Rate limiting.

Observability

  • Telemetry: OTelTelemetry wired to your existing TracerProvider. At minimum, surface loom.session.duration_ms, loom.tokens.input/output, loom.cost.usd, loom.budget.exceeded, loom.auto_extract.duration_ms, loom.auto_extract.invocations (last two appear when auto_extract is on), or the GenAI gen_ai.client.inference.* and gen_ai.execute_tool.duration metrics if your backend charts those. Metrics carry no user_id or session_id tags (unbounded cardinality); per-session breakdowns come from the spans’ session_id, per-user ones from the audit log. Dashboards keyed on OTel span names should use chat {model} / execute_tool {tool} / invoke_agent, or the loom.span.name attribute.
  • Audit log: FileAuditLog with a real HMAC secret. Every tool call and run-lifecycle transition lands here, attributed by user_id (top-level field; HMAC includes it).
  • Streaming: expose stream() so a UI / log pipeline can follow the loop in real time.
  • Multi-tenant load test: run bench/multi_tenant.py before any release that touches the agent loop, memory, or budget. Catches isolation regressions that unit tests miss.

Security

  • Permission policy: StandardPermissions(mode=Mode.DEFAULT) for interactive use; BYPASS only in CI / sandbox. For per-tenant policy routing, use PerUserPermissions(policies=, default=).
  • Approval handler: when destructive tools live behind PermissionDecision.ask_(...), wire Agent(approval_handler=callable) so the gate routes to a human, Slack, or a ticket queue. Without one, ask falls back to deny; it is never silently allowed. Return True, an action name ("allow", "deny", …) or an ApprovalDecision; anything else ("yes", {"approved": True}) denies. (Before 0.17 any truthy value approved.) See Approval handlers.
  • Filesystem sandbox: wrap any tool that touches the FS. Declare the allowed roots explicitly. See Sandboxes.
  • Pre-tool hooks: @agent.before_tool for any tool that sends external messages (email, Slack, etc.). A hook that raises or runs past its timeout denies the call.
  • Secrets: Agent(tuning=Tuning(secrets=EnvSecrets())) is the default; for vault-backed lookup pass a custom Secrets adapter. Use secrets.redact(text) before logging tool args / payloads so API keys don’t leak into the audit log.

Memory

  • Embedder: a real one (OpenAIEmbedder, VoyageEmbedder, CohereEmbedder, or a custom embedder) for production. HashEmbedder is for tests and zero-key dev only.
  • Auto-consolidate: Agent(..., tuning=Tuning(auto_consolidate=True)) if you want facts extracted automatically. Otherwise call await agent.consolidate() on a cadence.
  • Fact store: explicit (with_facts=True on the memory factory, or pass fact_store=...). Don’t rely on the in-memory default in production.

Testing

  • Test with ScriptedModel for deterministic multi-turn scenarios. EchoModel for the simplest smoke tests.
  • Fake the embedder with a small class of your own that maps specific texts to specific vectors when you need to assert on ranking.
  • Use the in-memory backends in tests (InMemoryMemory, InMemoryFactStore, InMemoryAuditLog, InMemoryJournalStore) so tests are fast and hermetic.
  • Skip live-integration tests with env-var gates, for example @pytest.mark.skipif(not os.environ.get("TEST_PG_DSN"), reason="no Postgres").
Last updated on
Streaming UI integrationOverview