Month: November 2025

Introduction

We often talk about training large language models (LLMs) as if it’s one thing — but it really isn’t.

There are several distinct types of training, each with its own purpose, cost, and level of control.

Understanding the difference helps clarify what’s realistic to do in practice, and what should be left to the model labs with thousands of GPUs and power budgets larger than small towns.

Here’s a simple breakdown.

Base Training — Learning Language from Scratch

This is where it all begins.

The model learns to predict the next word in a sentence across trillions of examples. It’s how it develops a general understanding of language, reasoning, facts, and relationships between concepts.

Purpose: Build a general-purpose foundation.

Data: Huge, diverse datasets (Common Crawl, Wikipedia, code, books).

Cost: Astronomical — only done by major labs.

Analogy: Teaching a child how to speak and read.

Once complete, this produces what’s called a base model — capable, but not polite, safe, or even particularly helpful.

Post-Training — Teaching Behavior and Alignment

After base training, the model needs to learn how to behave.

This phase adjusts it to follow instructions, respond helpfully, and align with human preferences and safety policies.

It typically involves:

• Supervised Fine-Tuning (SFT): The model learns from curated examples of correct input/output pairs.
• Reinforcement Learning from Human Feedback (RLHF): Humans rank several model responses, and the model learns to prefer the higher-ranked ones.
• Reinforcement Learning from AI Feedback (RLAIF): The same, but with AI systems acting as evaluators.

Purpose: Make the model cooperative and safe.

Analogy: Teaching manners, ethics, and social intelligence after language is learned.

All commercial models — GPT-4, Claude 3, Gemini, Llama 3 — go through this step before they ever reach users.

Fine-Tuning — Specializing for a Domain

Fine-tuning takes a general model and teaches it domain-specific knowledge: medicine, law, brewing, internal documentation — whatever your niche may be.

There are a few variants:

• Full fine-tuning: retraining all model weights (rare and expensive).
• Parameter-efficient fine-tuning (LoRA, QLoRA, PEFT): training small adapter layers on top of the frozen base model — vastly cheaper and reversible.

Purpose: Adapt to a domain or style.

Analogy: Sending a fluent speaker to medical school.

In practice, fine-tuning makes sense only if you have high-quality, well-structured data and a clear purpose — for example, improving factual recall in a specific knowledge domain or matching a company’s tone of voice.

Reinforcement Fine-Tuning — Teaching Preferences and Optimization

A newer development combines reinforcement learning with fine-tuning to optimize specific, measurable outcomes — such as factuality, brevity, or computational efficiency.

OpenAI’s Reinforcement Fine-Tuning (RFT) of the o1 model is one recent example: instead of relying on humans to rate outputs, the process automatically scores model responses using well-defined reward functions.

Purpose: Optimize behavior using measurable rewards.

Analogy: Practicing until performance metrics improve, rather than memorizing answers.

Retrieval-Augmented Generation (RAG) — Adding Knowledge Without Training

RAG isn’t training at all — it’s a retrieval technique.

The model stays frozen but is connected to a search index, database, or vector store.

When asked a question, it first retrieves relevant information and then generates an answer grounded in that content.

Purpose: Keep models current and connected to external knowledge.

Analogy: Looking something up rather than memorizing it.

RAG is ideal when data changes frequently, or when you can’t or shouldn’t embed sensitive data into the model itself.

Prompt Tuning — Lightweight Personality Shaping

The lightest-weight form of adaptation is prompt tuning, sometimes called soft prompting.

Here, a small vector (or a few tokens) is trained to steer the model’s behavior without modifying its core weights.

Purpose: Adjust tone or persona without retraining.

Analogy: Giving the same person a new job description — “today you’re the legal assistant.”

Prompt tuning is useful when you want to offer multiple personalities or roles from a single model.

What’s Reasonable to Do (and What Isn’t)

To make this practical, here’s how the different kinds of training align with the Four Categories of AI Solutions from simple Copilot-style automation to custom AI systems built from scratch.

Goal	Technique	Cost & Effort	Category (from “Four Categories of AI Solutions”)	Typical Use
General chatbot or Copilot	None (use aligned base model)	🟢 Low	Category 1 — Copilot / built-in AI features	Office copilots, internal Q&A bots
Domain expertise	LoRA / adapter fine-tuning	🟠 Medium	Category 2 — Configured / composable solutions	Industry copilots, internal assistants
Keep knowledge fresh	RAG or hybrid RAG + fine-tuning	🟠 Medium	Category 3 — Integrated or extended AI systems	Research assistants, customer-facing search
Optimize measurable output	Reinforcement fine-tuning	🔴 High	Category 4 — Custom AI / in-house LLMs	Scientific computing, advanced R&D
Create new model family	Base training	🚫 Extreme	Beyond Category 4	Reserved for foundation model labs

Summary

• Base training — teaches language.
• Post-training (SFT/RLHF) — teaches behavior.
• Fine-tuning — teaches domain knowledge.
• Reinforcement fine-tuning — teaches optimization.
• RAG / Prompt tuning — extend without retraining.

LLMs aren’t trained once — they’re trained in layers.

Each layer shapes a different aspect of intelligence: from raw linguistic intuition to helpful conversation, domain expertise, and ongoing adaptability.

Knowing which layer you’re working with isn’t just a technical detail — it’s the difference between using AI and building with it.