Example: evaluate a RAG agent

This example builds a small RAG agent that answers questions about a text file, records it in Studio, scores it on a handful of questions, then changes it and compares the two runs. It’s the loop Studio is for: see what the agent did, measure it, change it, check the change helped.

Setup


pip install 'loomflow[openai,chroma,studio]' python-dotenv

Put your OpenAI key in a .env file next to the script (OPENAI_API_KEY=sk-...) and the document in docs/game_of_thrones.txt. Any text file works; change the questions below to match it.

The agent and its eval

Save this as rag.py:


import asyncio
from pathlib import Path
 
from dotenv import load_dotenv
 
import loomflow.studio
from loomflow import Agent
from loomflow.eval import (
    Case, Contains, Dataset, EvalHarness, LLMJudge, ToolSelectionAccuracy,
)
from loomflow.loader import RecursiveChunker, load_text
from loomflow.memory.embedder import OpenAIEmbedder
from loomflow.model import OpenAIModel
from loomflow.tools import tool
from loomflow.vectorstore import ChromaVectorStore
 
load_dotenv()                 # OPENAI_API_KEY from .env
loomflow.studio.enable()      # record every run to .loomflow/studio.db
 
DOC_PATH = Path("docs/game_of_thrones.txt")
embedder = OpenAIEmbedder(model="text-embedding-3-small")
_store: ChromaVectorStore | None = None
 
 
async def get_store() -> ChromaVectorStore:
    """Open the vector store once; index the document the first time."""
    global _store
    if _store is None:
        store = ChromaVectorStore(
            embedder=embedder, collection_name="got", persist_directory=".chroma",
        )
        if await store.count() == 0:
            doc = load_text(DOC_PATH)
            chunker = RecursiveChunker(chunk_size=1000, chunk_overlap=100)
            chunks = chunker.split(doc.content, source=DOC_PATH.name)
            ids = [f"{DOC_PATH.stem}-{i}" for i in range(len(chunks))]
            await store.add(chunks, ids=ids)
        _store = store
    return _store
 
 
@tool
async def retrieve_data(query: str) -> list[str]:
    """Search the document for passages about the question."""
    store = await get_store()
    return [hit.chunk.content for hit in await store.search(query, k=4)]
 
 
agent = Agent(
    "You are a RAG agent. Call `retrieve_data` with the user's question, "
    "then answer only from the passages it returns.",
    name="pdf-rag",
    model=OpenAIModel(model="gpt-4.1-mini"),
    tools=[retrieve_data],
)
 
# Each expected answer is a short phrase the right answer must contain.
dataset = Dataset([
    Case(id="topic", input="What is this document about?",
         expected="Game of Thrones", expected_tools=["retrieve_data"]),
    Case(id="creators", input="Who created the series?",
         expected="Benioff", expected_tools=["retrieve_data"]),
    Case(id="source", input="Which novels is it based on?",
         expected="A Song of Ice and Fire", expected_tools=["retrieve_data"]),
    Case(id="seasons", input="How many seasons did it run?",
         expected="eight seasons", expected_tools=["retrieve_data"]),
], name="got-faq")
 
metrics = [
    Contains(case_sensitive=False),              # the answer contains the expected phrase
    ToolSelectionAccuracy(),                     # it called retrieve_data
    LLMJudge(OpenAIModel(model="gpt-5-mini")),   # another model grades the answer
]
 
 
async def main() -> None:
    print((await agent.run("What is this document about?")).output)
    report = await EvalHarness(agent, metrics).run(
        dataset, experiment="baseline", max_cost_usd=1.0,
    )
    print(report.summary())
 
 
if __name__ == "__main__":
    asyncio.run(main())

What matters in it:

The tool takes the question (query: str), so the agent searches for what was asked. A tool with no arguments that always searches the same text returns the same passages for every question, and no eval can tell good answers from bad ones.
The store opens once and indexes the document only the first time, into .chroma/.
The agent has a name=, so Studio keeps its versions together as you edit it.
Expected answers are short phrases, like Benioff, that a right answer must contain. expected_tools says the agent should have called retrieve_data.
Everything runs in one main(), under if __name__ == "__main__":, so importing the file (as loom studio --agent does) runs nothing.

Run it


python rag.py

It prints the agent’s answer to one question, then the eval’s summary: each metric’s mean, lowest score and number of cases scored.


{'contains': {'mean': ..., 'min': ..., 'count': 4.0},
 'llm_judge': {'mean': ..., 'min': ..., 'count': 4.0},
 'tool_selection_accuracy': {'mean': ..., 'min': ..., 'count': 4.0}}

Look at it in Studio


loom studio

Experiments → got-faq → baseline lists each question with its expected answer, the agent’s answer and its scores.
Open a case’s trace to see why: the retrieve_data call with the query it searched and the passages it got back, then the model call that wrote the answer from them.
Agents → pdf-rag draws the agent with its tool, and lists its versions.

Reading the scores

Metric	Low means
`contains`	The answer doesn’t contain the expected phrase. Either it’s wrong, or the expected phrase is too long or worded differently.
`tool_selection_accuracy`	The agent answered without searching: tighten the instructions.
`llm_judge`	The judge model thinks the answer is wrong or doesn’t match the reference.

contains 0, llm_judge 1? The expected answer is probably a whole paragraph, for example the run’s own output kept as it was when you added it to a dataset. A model never writes the same paragraph twice, so contains fails while the judge, which reads for meaning, passes. Keep expected answers to a short phrase, or score long answers with the judge alone.

Change it and compare

Change the agent, give the run a new name, and run it again. For example, make the instructions stricter:


agent = Agent(
    "You are a RAG agent. Always call `retrieve_data` with the user's question first. "
    "Answer in one sentence, using only the passages it returns.",
    name="pdf-rag",
    model=OpenAIModel(model="gpt-4.1-mini"),
    tools=[retrieve_data],
)

and in main(), experiment="one sentence". Run python rag.py again. Studio records the edited agent as pdf-rag v2, with a note that its instructions changed. On Experiments, pick baseline and one sentence and choose Compare. Each question is marked improved, regressed or unchanged, and the cost and time of the two runs are compared.

Run it from the Studio page instead

Studio can run the eval itself:

Start Studio with the agent


loom studio --agent rag:agent

Run it from the folder that holds rag.py, with your key in .env.

Build a dataset from real runs

Open a run on Traces, choose Add to dataset…, and replace the expected output with a short phrase the right answer must contain. Name the dataset, for example got-from-runs. Repeat for a few runs.

Run the eval

On the dataset’s page, choose Run eval. Pick pdf-rag, tick Contains and LLM judge (judge model gpt-5-mini), keep the spending limit, and Start eval. The experiment’s page updates as it runs and links to Compare with the last run when it’s done.

Studio imports rag.py when it starts, so after you edit the agent, restart loom studio before running the eval again.

Mistakes this example avoids

Two asyncio.run(...) calls. The OpenAI client belongs to the first event loop, so the second call can fail with “Event loop is closed”. Await everything inside one main().
Work at import time. loom studio --agent rag:agent imports the file; a module-level asyncio.run(...) would run your eval right then.
A string where a Path is needed. DOC_PATH.name and DOC_PATH.stem work on a Path, not on a plain string.
The agent’s own model as the judge. A model tends to agree with itself; use a different one. The harness warns if a judge is the agent’s own model object.