The Studio web app

Running it

pip install 'loomflow[studio]'
loom studio                                # reads .loomflow/studio.db
loom studio --db path/to/studio.db --port 7600 --open

The app reads the file while your app writes it, so you can leave it open and reload to see new runs. If the file doesn’t exist yet, the page tells you how to start recording.

Pages and URLs

Every view is a plain URL, so you can bookmark or share a filtered list or a single step.

Scoring runs and building datasets

A trace’s page has a Scores bar under the run’s summary. It shows the run’s scores as chips (× removes one, wherever it came from) and has:

• Good and Bad: score the run in one click, under the key review.
• Other score…: any key (keys already in use are suggested), with Good, Bad, a number or a label, and an optional comment.
• Add to dataset…: the run’s input and output side by side, ready to edit. Change the output into the answer the agent should have given, pick a dataset (the last one you used is filled in; a new name creates one) and add the case.

Scores you add on the page are marked ui.

The Datasets page lists the datasets with their cases, and a JSONL download; loomflow.studio.dataset("name") loads one for the eval harness (Scores and experiments).

To work through runs, tick Not scored on the Traces page: it lists the runs nobody has scored yet.

Running evals from the page

Start Studio with the agents it may run, and the page can run an eval on a dataset and record it as an experiment:

loom studio --agent app.agents:billing --agent app.agents:support

or list them once in pyproject.toml, and run plain loom studio:

[tool.loomflow.studio]
agents = ["app.agents:billing", "app.agents:support"]

Then use Run eval on a dataset’s page, Evaluate on a dataset on an agent’s page, or Run an eval on the Experiments page. The form picks:

• the agent (each shows what one of its runs has cost on average) and the dataset;
• how to score each case: the output contains the expected answer (ignoring case), matches it exactly, or an LLM judge’s score, with the judge model you name (for example gpt-5-mini);
• a name, a spending limit (it stops starting cases once the runs have cost that much; empty for none) and how many cases run at once.

The experiment’s page opens at once and refreshes itself while the eval runs, with a Cancel button. When it ends it links to Compare with the previous run of the same dataset. One eval runs at a time.

Architecture diagrams

An agent’s page opens with a diagram of the version you’re looking at, with a tab for each version so you can see how the structure changed. A trace’s page shows the diagram of the version that ran, with the parts that run used highlighted and the rest dimmed.

What the diagram shows:

• Agents by their name= (or the first line of their instructions), with the team’s architecture under the name (supervisor, router, swarm, …) and the tools each one holds.
• Teams: a supervisor’s delegate and forward edges to its workers, a router’s routes, an actor and critic loop, debaters and their judge, a blackboard. Edges that loop back up are dashed; a swarm whose every member can hand off to every other gets a note on its group instead of a tangle of lines.
• Workflows: start, each step (a function, an agent or a nested workflow), routes labelled with their keys, parallel branches, and end.

Diagrams are drawn as SVG on the server, like the rest of the app, with no JavaScript. Under each one, As text lists the same connections for screen readers or copying.

Mounting it in another app

create_app returns a plain ASGI app, like loomflow.serve, so it runs under any ASGI server or mounts inside a Starlette or FastAPI app:

from loomflow.studio.web import create_app
 
studio_app = create_app(
    ".loomflow/studio.db",
    read_only=False,                         # True hides the forms
    allowed_hosts=["studio.internal"],       # the name you reach it by
    agents=["app.agents:billing"],           # or Agent objects: evals from the page
)

The app answers only to the host names in allowed_hosts (by default localhost, 127.0.0.1, ::1 and any *.localhost), so when you serve it under another name, list that name; ["*"] answers to any.

Keeping it private

The app itself is careful with what it serves and accepts:

• It changes only what you add from its pages, scores and dataset cases, and never your recorded runs. --read-only turns even that off.
• A form is accepted only with a token from the app’s own pages, and refused when the browser says it came from another site, so a web page you visit can’t post to Studio.
• It answers only to its own host names, so a web page can’t read it by pointing a name of its own at 127.0.0.1. loom studio --host adds the address it listens on; listening on 0.0.0.0 answers to any name, and loom studio warns you that anyone who can reach the port can read your traces.
• It serves no JavaScript, every recorded value is HTML-escaped, and its pages carry a Content-Security-Policy that forbids scripts and outside requests, so a prompt containing markup can’t run in your browser.