Pre-training
In one sentence
The stage where a large language model learns language and general knowledge from massive volumes of public text, determining its baseline capabilities and knowledge cutoff date.
Pre-training requires massive amounts of data and compute, typically done only by model providers. Once pre-training is complete, the model's knowledge stops at that cutoff date; it does not know about subsequent news or new products.
To help a model understand their business, enterprises generally do not pre-train from scratch; instead, they use knowledge bases, RAG, or fine-tuning.
Example
If you ask a model, "What is the latest smartphone released this year?" and it answers with information from before its training cutoff, that is the pre-training knowledge cutoff in action.
Related terms
- Fine-tuning
- Training an already pre-trained foundation model further on a company's own dataset so it adapts to a specific task or writing style.
- Large Language Model (LLM)
- AI models trained on massive volumes of text that can understand and generate text. Doubao, DeepSeek, ChatGPT, and Claude are all powered by large language models.
- RAG (Retrieval-Augmented Generation)
- A method where AI retrieves relevant information from a knowledge base first, then generates an answer based on those materials.
