Recently Added
3,528 documents
Kerchunk
[kerchunk][kerchunk] supports cloud-friendly access of data
Document Chunking/Splitting in Langroid
Langroid's [`ParsingConfig`][langroid.parsing.parser.ParsingConfig]
libragen: First-Class AST-Aware Code Chunking Support
**Status:** In Progress
How-to: Customize Chunking Strategies
This guide shows you how to choose and configure different chunking strategies for your RAG pipeline. You can read more about why chunking matters in [Explanation: Understanding Chunking Strategies](../explanations/understanding_chunking.md).
Chunking Strategies
LLMs have context limits. You can't pass an entire 200-page SEC filing to an LLM for entity extraction. Documents must be broken into smaller pieces—**chunks**—that fit within processing limits.
语义块切分 Semantic Chunking
文本块切分是检索增强生成（RAG）中的关键步骤，其中将大文本体分割成有意义的段落以提高检索准确性。与固定长度块切分不同，语义块切分是根据句子之间的内容相似性来分割文本的。
Chunking Strategies
Chunking strategies are critical for dividing large texts into manageable parts, enabling effective content processing and extraction. These strategies are foundational in cosine similarity-based extraction techniques, which allow users to retrieve only the most relevant chunks of content for a given query. Additionally, they facilitate direct integration into RAG (Retrieval-Augmented Generation) systems for structured and scalable workflows.
🧠 Adaptive RAG Chunking System
The original system was creating too many tiny chunks (14 chunks for 1793 characters), fragmenting context and reducing answer quality. The new **adaptive chunking system** intelligently handles all document types with optimal chunk sizes.
rag-vector-chunking
The architecture you described is the standard setup for embedding-based retrieval in RAG systems:
With Jina Embeddings v3
Late chunking is a technique where you embed the entire document first with a long-context embedding model, then chunk the resulting contextualized representations, rather than chunking text first then embedding each chunk independently.
Late Chunking Strategy
**Late Chunking** is an advanced method for preparing long documents for retrieval systems, designed to overcome the critical problem of context loss that occurs in traditional document processing.
Step 7: Action Chunking
Instead of predicting one action at a time, predict a sequence of actions (chunk). This captures temporal structure and is a key idea from ACT (Action Chunking with Transformers) that carries into Pi0.
ADR-0006: Paragraph-Boundary Chunking
**Date:** Retroactive
Chunking
This RFC proposes a modification to the Kimchi proof system and the pickles recursion layer to increase the circuit size limit by splitting the polynomials from a circuit into 'chunks' which are less than the hard limit of 2^16 that Mina / SnarkyJS supports.
Running Map Chunking on VPS
This guide explains how to run the map chunking process on your VPS server.
/tmp/example-workspace/WORKSPACE.bazel:13:13
title: Bazel WORKSPACE chunking
Message Chunking for MCP stdio Transport
When using stdio pipes for MCP communication on macOS, there's a 64KB limit on the amount of data that can be written to a pipe in a single operation. This limitation can cause issues when sending large JSON-RPC messages, such as tool responses with substantial data or resource contents.
20. Text Flushing and Chunking (Current Behavior)
This document explains the *current* message flushing/chunking pipeline used by TomoriBot when streaming model output to Discord.
Haskell ASTChunking Enhancement
This document describes the comprehensive enhancement of Haskell code chunking in CocoIndex, inspired by techniques from the ASTChunk library. The improvements transform the original basic regex-based approach into a sophisticated, configurable chunking system with rich metadata and intelligent boundary detection.
Document_Processing_Chunking
ProcessorBase["ProcessorBase"]
ADR: Preserve Original Memories During Chunking
*Status: Accepted – 2025-01-27*
Fix Summary: Matroska Adaptive Chunking for 128D Embeddings
**Critical Bug**: [clustering_rpn.py:43-48](knowledge3d/cranium/clustering_rpn.py#L43-L48) was truncating 128-dimensional embeddings to **4 dimensions**:
Enable chunking with 8 chunks (the default threshold of 8192 tokens)
title: Row-Parallel Chunking
File Chunking Configuration
To prevent timeouts when reviewing large files, the tool automatically splits files into manageable chunks based on function boundaries.