Context window
In one sentence
The maximum total amount of text an LLM can process at once, including prompts, reference materials, chat history, and its own generated responses.
Context windows are measured in tokens. Any content exceeding this limit becomes invisible to the model. Even within the limit, if the content is too long, the model's attention on the middle section diminishes, making it prone to missing or misremembering details.
Therefore, when dealing with large volumes of data, it is not recommended to feed all files to the model at once. Instead, retrieve the relevant passages first and then pass them to the model—which is how RAG works.
Example
Pasting 20 contracts into an AI at once for comparison and having it mix up the amount from contract #7 with contract #12 is often caused by an excessively long context.
Related terms
- Token
- The basic unit LLMs use to process text. Both model context length and API call costs are calculated in tokens.
- RAG (Retrieval-Augmented Generation)
- A method where AI retrieves relevant information from a knowledge base first, then generates an answer based on those materials.
- Long-term memory
- Information an AI retains across multiple conversations, such as user preferences, past decisions, and completed actions.
- Hallucination
- When an AI presents non-existent or incorrect information with full confidence, such as fabricating data, sources, legal clauses, or product features.
