About semchunk
semchunk is a Python library designed to split text into smaller chunks while preserving as much local semantic context as possible. It supports AI-powered chunking, chunk overlapping, and chunk offsets, and works seamlessly with any tokenizer or token counter, including those from Tiktoken and Transformers. Powered by a novel hierarchical chunking algorithm, semchunk claims to deliver 15% better RAG performance compared to its closest competitors. The library is production-ready, with millions of monthly downloads, and is used in projects like Docling, the Microsoft Intelligence Toolkit, and the Isaacus API. Installation is available via pip, uv, and conda-forge, and a Rust port (semchunk-rs) is maintained by a community member.
Key Features
AI-powered chunking with semantic preservation
Chunk overlapping and chunk offsets
Works with any tokenizer or token counter (Tiktoken, Transformers, custom)
Novel hierarchical chunking algorithm that improves RAG performance by 15%
Simple chunkerify() interface for quick integration
Multiprocessing support for batch processing
Production-ready with millions of monthly downloads
Available via pip, uv, and conda-forge; also includes a community-maintained Rust port
Pros & Cons
Pros
- Preserves semantic context better than simple token splitting
- Highly efficient with multiprocessing support for large text volumes
- Works with any tokenization scheme, giving flexibility
- Production-tested with millions of downloads per month
- Easy to use via the chunkerify() helper function
Cons
- AI-powered chunking requires an Isaacus API key and additional SDK (isaacus)
- Only available as a Python library (official), though a community Rust port exists
- Chunk size must account for special tokens added by the tokenizer, which may require manual adjustment
Best For
Preprocessing text for Retrieval-Augmented Generation (RAG) pipelinesChunking documents for LLM context windowsSplitting long texts for semantic search or indexingData preparation for NLP model training or inference
FAQ
What tokenizers does semchunk support?
semchunk works with any tokenizer or token counting function. It supports OpenAI models (e.g., gpt-4), Tiktoken encodings (cl100k_base), Hugging Face Transformers tokenizers, and custom callable functions via the chunkerify() method.
How is semchunk installed?
semchunk can be installed via pip or uv: 'pip install semchunk'. It is also available on conda-forge: 'conda install conda-forge::semchunk'. For AI-powered chunking, additionally install the Isaacus SDK: 'pip install isaacus'.
Does semchunk work with any programming language?
The official library is for Python, but there is a community-maintained Rust port called semchunk-rs by dominictarro.