Purpose
This article explains the major training and adaptation stages used in modern LLM development: large-scale pre-training, continued pre-training, supervised fine-tuning, instruction tuning, and parameter-efficient fine-tuning techniques such as LoRA.
Overview
A production LLM is rarely the result of a single training phase. In most practical pipelines, the model passes through several stages, each optimizing for a different objective:
- pre-training to learn a broad generative prior over language and code
- continued pre-training to adapt to a new domain or data distribution
- supervised fine-tuning (SFT) to shape task behavior
- instruction tuning to improve instruction following and assistant-like responses
- preference optimization or RL-based post-training to align outputs with preferred behavior
- optional PEFT-based specialization for domain- or customer-specific adaptation
The distinction between these phases matters because the data format, loss function, optimizer settings, parameter update scope, and failure modes differ substantially across them.
Generative pre-training
Pre-training is usually an autoregressive language-modeling phase over very large text and code corpora.
For a tokenized sequence x_1, ..., x_T, the training objective is:
maximize sum_t log p(x_t | x_<t))
Equivalently, minimize cross-entropy between predicted token distributions and the true next token.
This objective looks simple, but it is extremely information-dense. To minimize loss effectively, the model must internalize:
- lexical and syntactic regularities
- semantic associations
- discourse patterns
- common document structures
- latent task formats such as question-answering, summarization, explanation, and code completion
The model is not explicitly told “this is translation” or “this is reasoning.” Those behaviors emerge because the next-token objective incentivizes modeling the generative structure of many tasks present in the corpus.
Tokenization matters
The pre-training objective is defined over tokens, not raw characters or words. Therefore tokenizer design affects:
- sequence length
- vocabulary size
- efficiency on code vs natural language
- multilingual behavior
- compression of repeated patterns
Tokenization choices indirectly affect compute efficiency because the total training budget is typically measured in tokens processed, not documents.
Compute, data, and scaling laws
Model quality does not scale with parameter count alone. It depends jointly on:
- parameter count
- number of training tokens
- optimization budget / FLOPs
- data quality and deduplication
- architecture efficiency
A central result from scaling-law work is that under a fixed compute budget, there is a compute-optimal tradeoff between model size and training tokens. A model that is too large for its data budget will be undertrained; a model that is too small for the available compute will underuse the budget.
This is why a “larger model” is not automatically a “better model.” In practice, strong training pipelines balance:
- parameter count
- token budget
- learning-rate schedule
- batch size and optimizer stability
- curriculum and mixture design
Data quality is not optional
At LLM scale, data filtering and deduplication are first-order concerns. Repeated low-quality or near-duplicate content can:
- waste compute
- distort loss estimates
- increase memorization risk
- reduce generalization quality
Pre-training quality depends heavily on the data pipeline, not just the optimizer or model shape.
Continued pre-training
Continued pre-training extends a pretrained model using more language-model-style training after the initial foundation stage.
Typical motivations:
- ingest a new domain corpus
- improve performance on internal terminology or document style
- shift toward a different language mix
- refresh knowledge up to a later cutoff
- adapt a general model toward code, scientific text, legal text, or enterprise documentation
This stage still uses the language-model objective, so it preserves the generative prior more naturally than narrow task fine-tuning.
However, continued pre-training must be managed carefully. If the new corpus is too narrow or too aggressively weighted, the model can drift:
- overfit to the new domain style
- lose performance on general tasks
- become imbalanced in tone or vocabulary
This is essentially a distribution-shift problem.
Supervised fine-tuning (SFT)
SFT trains the model on labeled input-output pairs:
- prompt -> desired response
- document -> summary
- instruction -> structured output
- conversation history -> assistant reply
Unlike pre-training, SFT is not trying to model broad internet-scale language statistics. It is trying to shape conditional behavior more directly.
Formally, it is still usually a token-level cross-entropy objective, but now conditioned on curated examples rather than raw next-token streams.
What changes in SFT
Compared with pre-training, SFT usually involves:
- much smaller datasets
- lower learning rates
- fewer optimization steps
- more aggressive monitoring of validation behavior
- stronger risk of overfitting or capability drift
Because the dataset is curated, SFT can quickly reshape behaviors such as:
- output format compliance
- instruction following
- response style
- structured extraction
- domain-specific terminology usage
Common failure modes
SFT is powerful, but brittle if done poorly. Common problems include:
- catastrophic forgetting of general capability
- style overfitting
- poor calibration outside the supervised distribution
- spurious improvement on benchmark-like evals that does not transfer to real use
This is why evaluation must include both task-specific and broad regression checks.
Instruction tuning
Instruction tuning is a specific form of SFT in which the training corpus is dominated by instruction-response examples.
The goal is not just correctness, but compliance with the interaction pattern:
- identify the requested task
- infer the desired level of detail
- answer in the requested format
- avoid ignoring the instruction frame
This is one reason raw base models and instruction-tuned models can feel dramatically different even if they share the same pretrained backbone.
Data sources
Instruction-tuning datasets can be built from:
- human annotation
- synthetic expansion
- self-instruct pipelines
- distillation from stronger frontier models
- task reformulation from existing corpora
In practice, mixture design matters. Too much narrow templated instruction data can produce brittle assistant behavior that sounds aligned but lacks robustness.
Full fine-tuning
In full fine-tuning, all trainable model parameters are updated.
Advantages:
- maximum adaptation capacity
- strongest distribution shift if enough data exists
- no architectural constraints from adapter placement
Disadvantages:
- high optimizer-state memory cost
- expensive checkpoint storage for multiple variants
- risk of overwriting useful pretrained representations
- operational complexity for maintaining many tuned descendants
For very large models, full fine-tuning may require distributed training even for modest adaptation tasks.
Parameter-Efficient Fine-Tuning (PEFT)
PEFT methods update only a small subset of trainable parameters while keeping the backbone mostly or entirely frozen.
This has several practical benefits:
- lower memory usage during training
- smaller artifacts for deployment or sharing
- easier multi-tenant specialization
- faster experimentation on multiple downstream tasks
PEFT is not one method; it is a family of approaches that trade off flexibility, cost, and engineering complexity.
LoRA
LoRA (Low-Rank Adaptation) is one of the most widely used PEFT techniques.
Instead of directly updating a pretrained weight matrix W, LoRA keeps W frozen and learns a low-rank update:
W' = W + Delta WDelta W = B A
where:
A in R^(r x d_in)B in R^(d_out x r)ris a small rank, typically much smaller thanmin(d_in, d_out)
This means the trainable update is constrained to a low-rank subspace. In many practical cases, that is enough to adapt the model effectively.
Why LoRA works well
The intuition is that downstream adaptation often does not require full-rank movement of every weight matrix. A low-dimensional update can capture much of the useful task-specific shift.
Important LoRA hyperparameters
Operationally, LoRA behavior depends strongly on:
- rank
r - alpha / scaling
- dropout
- target modules (e.g.
q_proj,k_proj,v_proj,o_proj, MLP projections) - whether adapters are merged into base weights or kept separate at inference
Placement matters
Applying LoRA only to attention projections is cheaper but less expressive. Applying it also to MLP projections can improve adaptation at the cost of more parameters.
Prompt tuning and prefix-like methods
Prompt tuning keeps the backbone frozen and learns only a small trainable prompt representation in embedding space.
Variants include:
- soft prompt tuning
- prefix tuning
- P-tuning-like methods
These methods are lighter than LoRA, but usually offer less adaptation capacity for difficult domain shifts. They can still be useful when:
- training resources are extremely constrained
- task format is stable
- the goal is lightweight steering rather than deep specialization
Practical training concerns
Regardless of whether the adaptation is full FT or PEFT, the training system must still handle:
- effective batch size
- sequence packing
- truncation policy
- optimizer choice
- gradient clipping
- mixed precision
- gradient accumulation
- checkpointing and resume behavior
For long-context or instruction-tuning runs, data formatting is often as important as optimizer configuration. Small mistakes in chat-template formatting, separator tokens, label masking, or EOS placement can materially affect behavior.
Label masking
In assistant-style SFT, it is common to compute loss only on assistant response tokens and mask user/system tokens from contributing to the objective. This makes the training signal better match the intended behavior.
Packing vs non-packing
Sequence packing improves hardware efficiency by concatenating multiple shorter examples into one training sequence. However, if masking is implemented incorrectly, examples can leak into one another. Packing must therefore be handled carefully.
Variance and brittleness
Fine-tuning results can vary significantly with:
- random seed
- data ordering
- exact prompt formatting
- learning-rate warmup
- checkpoint selection
- validation protocol
This matters because downstream performance is often noisy, especially on small or narrow datasets. Early stopping and robust eval design are not optional.
Why this matters in practice
These concepts map directly to real platform configuration fields and operational decisions:
- model source / base checkpoint
- tokenizer compatibility
- learning rate
- batch size and accumulation
- number of epochs or max steps
- LoRA rank, alpha, target modules
- validation cadence
- output checkpoint format
- merge or non-merge deployment strategy
A team that treats all “fine-tuning” as one generic process will usually make poor choices. The correct setup depends on whether the goal is broad knowledge adaptation, assistant behavior shaping, domain specialization, or efficient tenant-specific customization.
Key takeaways
- Pre-training learns a general generative prior through large-scale next-token prediction.
- Continued pre-training adapts that prior to a new distribution while preserving the LM objective.
- SFT shapes conditional behavior using curated input-output pairs.
- Instruction tuning is SFT specialized for instruction-following interaction patterns.
- Full fine-tuning maximizes flexibility but is expensive.
- PEFT methods, especially LoRA, enable strong adaptation at much lower training and storage cost.
Comments
0 comments
Article is closed for comments.