blazechunk

the fastest semantic text chunking library

blazechunk splits text at semantic boundaries and does it stupid fast: a SIMD-accelerated Rust core with a small, uniform Python API. It ships nine chunkers, reads documents from PDF to Excel, and every high-level chunker offers matching synchronous and asynchronous methods with full type hints and docstrings.

📖 Documentation: https://blazechunk-documentation.vercel.app/

Features

• ⚡ SIMD-accelerated Rust core — the raw size-based chunking primitive reaches ~1 TB/s on a 32 KB chunk size (see Benchmarks for what that number does and does not cover; the high-level chunkers do more work and run ~1 GiB/s).
• 📄 Document input — DocumentChunker reads PDF, Word, PowerPoint, Excel, OpenDocument, RTF, EPUB and CSV, and every chunk knows which heading it came from (pip install "blazechunk[anydoc]").
• 🧩 Nine chunkers — a zero-copy byte Chunker plus RecursiveChunker, SentenceChunker, TokenChunker, TableChunker, CodeChunker, and the embedding-based SemanticChunker, SDPMChunker, and LateChunker.
• 🔁 Sync and async — every chunker has chunk / chunk_async and chunk_batch / chunk_batch_async; async work runs off the event loop.
• 🔤 Pluggable tokenizers — count by character, word, byte, or table row out of the box, or point at a HuggingFace tokenizer.json (with the hf-tokenizer build).
• 🔌 Framework integrations — drop-in adapters for LangChain and Agno (pip install "blazechunk[langchain]" / "blazechunk[agno]").
• 🧵 Typed & documented — ships py.typed and type stubs; every method has a docstring.

Installation

Install using pip

$ pip install -U blazechunk

Usage

Every high-level chunker exposes the same four methods, so once you know one you know them all.

from blazechunk import TokenChunker

chunker = TokenChunker(chunk_size=512, chunk_overlap=64)

# chunk a single document
for c in chunker.chunk("... a long document ..."):
    print(c.text, c.start_index, c.end_index, c.token_count)

# chunk many documents
batches = chunker.chunk_batch(["doc one ...", "doc two ..."])

Async

import asyncio
from blazechunk import RecursiveChunker

async def main() -> None:
    chunker = RecursiveChunker(chunk_size=2048)

    # await a single document — the work runs off the event loop
    chunks = await chunker.chunk_async("... a long document ...")

    # await many, with optional back-pressure
    batches = await chunker.chunk_batch_async(
        ["doc one ...", "doc two ..."], max_concurrency=8
    )

asyncio.run(main())

The nine chunkers

Chunker	Splits on
Chunker	byte-size windows at delimiter boundaries (zero-copy)
RecursiveChunker	a hierarchy: paragraphs → sentences → … → tokens
SentenceChunker	whole sentences, with optional overlap
TokenChunker	fixed-size token windows, with optional overlap
TableChunker	Markdown/HTML table rows (header repeated per chunk)
CodeChunker	structural code blocks (brace/indent aware)
SemanticChunker	semantic-similarity troughs between sentence windows
SDPMChunker	semantic + a skip-window double-pass merge
LateChunker	recursive boundaries + mean-pooled "late" embeddings

from blazechunk import SentenceChunker, TableChunker, CodeChunker

SentenceChunker(chunk_size=2048, chunk_overlap=128).chunk(prose)
TableChunker(chunk_size=3).chunk(markdown_or_html_table)
CodeChunker(chunk_size=2048, language="python").chunk(source_code)

Embedding-based chunkers

SemanticChunker, SDPMChunker, and LateChunker need vectors — and the pure-Rust core ships no model. You inject an embedder, exactly like you inject a tokenizer; the Rust orchestration calls back into it. Pass any callable embed_batch(list[str]) -> 2D, or an object exposing embed_batch / encode (e.g. sentence-transformers or model2vec):

from sentence_transformers import SentenceTransformer
from blazechunk import SemanticChunker, SDPMChunker, LateChunker

model = SentenceTransformer("all-MiniLM-L6-v2")

# SemanticChunker / SDPMChunker: one vector per sentence window.
semantic = SemanticChunker(model, threshold=0.8, chunk_size=2048)
chunks = semantic.chunk(prose)          # -> list[Chunk], partitions the text

# SDPM adds a skip-window second pass that re-merges related, non-adjacent groups.
sdpm = SDPMChunker(model, skip_window=1).chunk(prose)

# LateChunker embeds the whole document once, then mean-pools per chunk. It needs
# token-level embeddings: an object with embed_as_tokens(text) and embed(text)
# (or a (embed_as_tokens, embed) tuple). Each result is a LateChunk with `.embedding`.
late = LateChunker(token_model, chunk_size=2048).chunk(document)
vec = late[0].embedding                 # the chunk's late-interaction vector

Zero-copy fast path

The Chunker primitive and the chunk() helper yield zero-copy memoryview slices for maximum throughput:

from blazechunk import chunk

for view in chunk(b"Hello. World. Test.", size=10, delimiters=b"."):
    print(bytes(view))

Documents

Every RAG pipeline starts with a file, not a string. DocumentChunker takes the file — PDF, Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV — and returns chunks that carry the structure they came from.

pip install "blazechunk[anydoc]"

from blazechunk.loaders import DocumentChunker

result = DocumentChunker().chunk("report.pdf")

for c in result.chunks:
    print(c.heading_path, c.kind, c.text[:60])
    # ('Methods', 'Sample Preparation')  prose  'We sampled two hundred sites across …'

The document is segmented before it is chunked, so each piece is routed to a chunker that suits it — table rows to TableChunker (never split mid-row), fenced code to CodeChunker (fences intact), prose to whichever chunker you picked:

from blazechunk import RecursiveChunker, TableChunker
from blazechunk.loaders import DocumentChunker

loader = DocumentChunker(
    chunker=RecursiveChunker(chunk_size=2048),
    table_chunker=TableChunker(chunk_size=3),
    respect_headings=True,      # never merge across a heading boundary
)

result  = loader.chunk("handbook.docx")
result  = await loader.chunk_async("report.pdf")
results = loader.chunk_batch(paths, on_error="skip")   # skip unreadable files

heading_path is the highest-value field here: it makes a fragment from the middle of a long document self-locating, and a reranker can use it directly.

Offsets. md_start / md_end are byte offsets into result.markdown — the converted Markdown, returned alongside the chunks — and not into the original file. anydoc exposes no mapping back to source bytes, so page-level attribution for a PDF is not something this can honestly provide, and no approximation is shipped in its place.

Scanned PDFs have no text layer and need OCR, which anydoc does not do. They raise a named ScannedDocumentError rather than a puzzling "unsupported format". Conversion is handled by anydoc, a pure-Rust converter from Firecrawl with no ML and no network calls.

Integrations

blazechunk plugs into popular RAG frameworks — install the matching extra.

LangChain

pip install "blazechunk[langchain]"

from blazechunk import TokenChunker
from blazechunk.integrations.langchain import BlazechunkTextSplitter

splitter = BlazechunkTextSplitter(TokenChunker(chunk_size=512, chunk_overlap=64))
docs = splitter.create_documents([text])

Agno

pip install "blazechunk[agno]"

from blazechunk import TokenChunker
from blazechunk.integrations.agno import BlazechunkChunking

strategy = BlazechunkChunking(TokenChunker(chunk_size=512, chunk_overlap=64))
# TextKnowledgeBase(path="docs", vector_db=..., chunking_strategy=strategy)

Both adapters also provide async variants — LangChain asplit_text / atransform_documents and Agno achunk — so they run off the event loop in async ingestion pipelines.

Benchmarks

The headline number is the raw SIMD size-based chunking primitive — finding split offsets in a byte buffer — measured on enwik8/enwik9 (Wikipedia extracts) on an Apple Silicon MacBook. It is not end-to-end throughput, and it scales with chunk size because larger chunks mean fewer boundaries to find:

Input	Chunk size	Throughput
enwik8 (100 MB)	32 KB	1.7 TB/s
enwik8 (100 MB)	16 KB	680 GB/s
enwik8 (100 MB)	4 KB	190 GB/s
enwik9 (1 GB)	32 KB	1 TB/s
enwik9 (1 GB)	4 KB	50 GB/s

The five high-level chunkers do more work (tokenize + split + merge), so their throughput is naturally lower — reported per-chunker so the numbers stay honest:

Chunker	Input	Throughput
RecursiveChunker	1 MB prose	~1.0 GiB/s
TableChunker	2000-row markdown	~1.7 GiB/s
CodeChunker	~25 KB Rust source	~800 MiB/s
SentenceChunker	50 KB prose	~670 MiB/s
TokenChunker	50 KB prose	~640 MiB/s

Reproduce with cargo bench (see benches/README.md for the full table and methodology).

Contributing

blazechunk is open source and contributions are very welcome — a bug report, a new chunker, a performance win, or a docs fix all help.

• 🐛 Found a bug or want a feature? Open an issue.
• 🔧 Want to send a change? See CONTRIBUTING.md for the dev setup (Rust + maturin), how to run the tests, and the PR checklist.

By contributing you agree your work is dual-licensed under MIT and Apache-2.0, matching the project.

Acknowledgements

blazechunk is a fork of the excellent chonkie-inc/chunk project and builds on its SIMD chunking core.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.