Chunkers

A chunker splits a loaded document into pieces small enough to embed and to return as search results. Pick one by how the source text is structured. There are four strategies for breaking a Document.content into LLM-friendly chunks, and all of them produce a list[Chunk] whose metadata points back at the source.

One-liner

from loomflow.loader import load, chunk
 
doc = load("research.pdf")
chunks = chunk(doc.content)             # default: RecursiveChunker

Pick one explicitly

from loomflow.loader import load_pdf, MarkdownChunker
 
doc = load_pdf("research.pdf")
chunker = MarkdownChunker(chunk_size=800, chunk_overlap=100)
chunks = chunker.split(doc.content)

All chunkers share the same .split(text, *, source="") interface. Pass source= so the chunk’s metadata records its provenance:

chunks = MarkdownChunker().split(doc.content, source=str(doc.metadata["source"]))

Chunk shape

@dataclass
class Chunk:
    content: str
    metadata: dict[str, Any]            # source, index, chunk_size, headers, ...

metadata carries source (from the parent Document), index (zero-based chunk index), chunk_size (length of content in chars), strategy (which chunker produced it), and strategy-specific keys such as headers from MarkdownChunker or token_count from TokenChunker.

RecursiveChunker

Splits on a hierarchy of separators (paragraph → line → sentence terminators → word → character). Most general-purpose, handles arbitrary text well.

from loomflow.loader import RecursiveChunker
 
chunker = RecursiveChunker(
    chunk_size=800,
    chunk_overlap=50,
    separators=[                                 # default
        "\n\n", "\n", ". ", "! ", "? ",
        "; ", ", ", " ", "",
    ],
)

This is the same algorithm as LangChain’s RecursiveCharacterTextSplitter, so chunk boundaries behave the way you’d expect if you’re porting a pipeline from it. The default separators add sentence and clause terminators to LangChain’s list. As in LangChain, a chunk never exceeds chunk_size (unless a single piece that can’t be split further is larger), the overlap is made of whole pieces rather than a raw character tail that starts mid-word, no text is lost, and chunks are stripped of surrounding whitespace.

MarkdownChunker

Splits on # heading boundaries. Each chunk records the header trail in metadata: metadata["headers"] is the path of headers the chunk lives under. # lines inside fenced code blocks (shell or Python comments) are not treated as headings.

from loomflow.loader import MarkdownChunker
 
chunker = MarkdownChunker(chunk_size=800, chunk_overlap=100)
chunks = chunker.split(doc.content)
 
for c in chunks[:3]:
    print(c.metadata["headers"])      # e.g. ["Methods", "Pre-processing"]

Pair with the PDF / DOCX / Excel loaders. They produce well-structured markdown, and MarkdownChunker preserves that structure as metadata the LLM can reason about.

SentenceChunker

Splits on sentence boundaries (. , ? , ! ). Best for prose where keeping individual sentences atomic matters (factual QA, citation- backed answers).

from loomflow.loader import SentenceChunker
 
chunker = SentenceChunker(chunk_size=400, chunk_overlap=80)  # overlap in chars, rounded to whole sentences

A sentence longer than chunk_size is split further, so no chunk comes out larger than the setting.

TokenChunker

Chunks by token count via tiktoken. Useful when the model has a hard context budget and you want to pack chunks to a precise size.

from loomflow.loader import TokenChunker
 
chunker = TokenChunker(
    chunk_size=512,         # tokens, not chars
    chunk_overlap=64,
    encoding="cl100k_base", # the default (GPT-4 encoding)
)

Chunks are cut on character boundaries, so CJK text and emoji don’t come back with U+FFFD replacement characters where a token slice split a multi-byte character.

Requires tiktoken:

pip install 'loomflow[loader-token]'