Text Chunking
In one sentence
Splitting long documents into appropriately sized chunks before saving them to a knowledge base, making it easy to retrieve only the relevant sections.
If chunks are too large, retrieved passages contain too much irrelevant text; if they are too small, sentences get cut off, losing context. A common practice is chunking by headings and paragraphs, maintaining a slight overlap between adjacent chunks.
Structured content like tables and clauses should be kept intact as much as possible, rather than cut in the middle.
Example
A 50-page product manual split into 120 chunks by chapter and subheading, with each chunk tagged with its chapter title.
Related terms
- RAG (Retrieval-Augmented Generation)
- A method where AI retrieves relevant information from a knowledge base first, then generates an answer based on those materials.
- Vector (Embedding)
- Converting a block of text or an image into a series of numbers. Content with similar meanings gets similar numbers, making it possible to retrieve information by meaning.
- Vector Database
- A database that stores vectors and quickly finds semantically similar content, commonly used as a core component for RAG and knowledge base retrieval.
