Chunkers

Four strategies for breaking a Document.content into LLM-friendly chunks. All produce a list[Chunk] whose metadata points back at the source.

One-liner

from loomflow.loader import load, chunk
 
doc = load("research.pdf")
chunks = chunk(doc.content)             # default: RecursiveChunker

Pick one explicitly

from loomflow.loader import load_pdf, MarkdownChunker
 
doc = load_pdf("research.pdf")
chunker = MarkdownChunker(chunk_size=800, chunk_overlap=100)
chunks = chunker.split(doc.content)

All chunkers share the same .split(text, *, source="") interface. Pass source= so the chunk’s metadata records its provenance:

chunks = MarkdownChunker().split(doc.content, source=str(doc.metadata["source"]))

Chunk shape

@dataclass
class Chunk:
    content: str
    metadata: dict[str, Any]            # source, index, chunk_size, headers, ...

metadata carries source (from the parent Document), index (zero-based chunk index), chunk_size (length of content in chars), and strategy-specific keys such as headers from MarkdownChunker or token_count from TokenChunker.

RecursiveChunker

Splits on a hierarchy of separators (paragraph → line → sentence terminators → word → character). Most general-purpose, handles arbitrary text well.

from loomflow.loader import RecursiveChunker
 
chunker = RecursiveChunker(
    chunk_size=800,
    chunk_overlap=50,
    separators=[                                 # default
        "\n\n", "\n", ". ", "! ", "? ",
        "; ", ", ", " ", "",
    ],
)

Defaults match LangChain’s RecursiveCharacterTextSplitter, so chunks port across systems.

MarkdownChunker

Splits on # heading boundaries. Each chunk records the header trail in metadata, metadata["headers"] is the path of headers the chunk lives under.

from loomflow.loader import MarkdownChunker
 
chunker = MarkdownChunker(chunk_size=800, chunk_overlap=100)
chunks = chunker.split(doc.content)
 
for c in chunks[:3]:
    print(c.metadata["headers"])      # e.g. ["Methods", "Pre-processing"]

Pair with the PDF / DOCX / Excel loaders. They produce well-structured markdown, and MarkdownChunker preserves that structure as metadata the LLM can reason about.

SentenceChunker

Splits on sentence boundaries (. , ? , ! ). Best for prose where keeping individual sentences atomic matters (factual QA, citation- backed answers).

from loomflow.loader import SentenceChunker
 
chunker = SentenceChunker(chunk_size=400, chunk_overlap=1)  # 1-sentence overlap

TokenChunker

Chunks by token count via tiktoken. Useful when the model has a hard context budget and you want to pack chunks to a precise size.

from loomflow.loader import TokenChunker
 
chunker = TokenChunker(
    chunk_size=512,         # tokens, not chars
    chunk_overlap=64,
    encoding="cl100k_base", # gpt-4 / claude default
)

Requires tiktoken:

pip install 'loomflow[loader-token]'