Scores and experiments

Scoring a run

loomflow.studio.score attaches a score to a recorded run. Pass the RunResult (or its run_id), a key and a value:

import loomflow.studio
from loomflow import Agent
 
agent = Agent("You are the billing agent.", model="claude-haiku-5-5", name="billing-agent")
result = await agent.run("My card was charged twice", user_id="alice")
 
loomflow.studio.score(result, "resolved", True, source="judge")
loomflow.studio.score(result, "rating", 4, comment="clear answer", source="user")
loomflow.studio.score(result.run_id, "tone", "polite")

A bool is stored as 1 or 0 and shown as yes or no; a number as itself; a string as a label. source says where the score came from. Scores show on the trace and in the traces list, and the Agents page shows each version’s mean score.

OnlineScorer records its scores the same way, on the run it scored, so sampled live scoring shows up next to each run with no extra code.

You can also score a run on its trace page in loom studio: Good or Bad in one click, or a number or label under any key, with a comment, from Other score…. The Not scored filter on the Traces page lists the runs still to review.

Building a dataset from runs

When a run goes wrong, keep it as a test case. On its trace page, Add to dataset… offers the run’s input and output: edit the output into the answer the agent should have given, name a dataset, and add it. The Datasets page lists what you’ve collected.

Run an eval on it with loomflow.studio.dataset:

import loomflow.studio
from loomflow import Agent
from loomflow.eval import Contains, EvalHarness
 
agent = Agent("You are the billing agent.", model="claude-haiku-5-5", name="billing-agent")
dataset = loomflow.studio.dataset("billing-regressions")
report = await EvalHarness(agent, [Contains()]).run(dataset, experiment="confirm first")

It returns an eval Dataset named after the Studio dataset, so its experiments group together, and each case keeps its id, so they compare case by case. Pass project= when several projects have a dataset of that name, and path= to read another file. Or download the dataset’s JSONL from its page and use loom eval billing-regressions.jsonl --agent app:agent --studio.

Eval runs as experiments

With Studio on, an EvalHarness run is recorded as an experiment: each case’s run is a trace, linked to the case, with its metric scores.

import loomflow.studio
from loomflow import Agent
from loomflow.eval import Contains, Dataset, EvalHarness
 
loomflow.studio.enable()
agent = Agent("You are the billing agent.", model="claude-haiku-5-5", name="billing-agent")
dataset = Dataset.from_jsonl("billing-faq.jsonl")      # named "billing-faq"
report = await EvalHarness(agent, [Contains()]).run(dataset, experiment="billing v2")

From the command line:

loom eval billing-faq.jsonl --agent app:agent --studio

• experiment= names the run (default: the agent’s name).
• Experiments are grouped by dataset name. Dataset.from_jsonl uses the file name; pass Dataset(cases, name="...") otherwise.
• Cases are matched across runs by their id when you gave one, or else by their input and expected output (generated ids change every time a file is loaded). Give cases ids if you edit their inputs.
• Eval runs are kept apart from your other runs: they don’t appear in the traces list or in the Agents page’s run counts, but their scores count towards each version’s mean score.

Running an eval from Studio

Started with loom studio --agent app.agents:billing (or with the agents listed under [tool.loomflow.studio] in pyproject.toml), Studio runs evals itself: Run eval on a dataset’s page, pick the agent, how to score the cases and a spending limit, and the run is recorded as an experiment like any other. Its page shows the progress and links to Compare with the previous run. The web app has the details.

Comparing two runs

On the Experiments page, pick a baseline and another run of the same dataset and choose Compare. Studio shows:

• each metric’s mean in both runs, and the change;
• cost per case and median latency, as a percentage change (changes under 5% count as no change);
• every case with both scores, marked improved, regressed or the same, judged on the first metric (a case that errored counts as regressed). Filter to the changed or regressed cases, and open either run’s trace to see why.

A typical loop: change the prompt or model, run the eval again, compare it with the last good run, and read the traces of the cases that regressed before you ship.