Purpose
This article explains three major classes of LLM efficiency techniques: distillation, quantization, and pruning, with emphasis on what they change in the model and how they affect deployment.
Overview
LLMs are often bottlenecked by one or more of the following:
- weight memory footprint
- KV-cache memory footprint
- memory bandwidth
- per-token latency
- training or fine-tuning cost
- deployment cost per request
Distillation, quantization, and pruning all aim to improve efficiency, but they operate at different levels:
- distillation changes how a smaller model is trained
- quantization changes numeric representation
- pruning changes model structure by removing parameters or blocks
These techniques are frequently combined in production pipelines.
Distillation
Distillation transfers behavior from a larger teacher model to a smaller student model.
The student is trained not only on ground-truth labels, but on teacher-generated signals that contain richer information about the teacher’s learned function.
Standard knowledge distillation
Instead of training the student only on hard targets, the student is trained to match the teacher’s output distribution.
This is commonly done by minimizing divergence between:
- teacher logits / probabilities
- student logits / probabilities
Often with a temperature parameter to soften the teacher distribution.
Why it helps:
- the teacher’s non-argmax probabilities encode similarity structure between outputs
- the student learns more than just “the correct label”
Sequence-level distillation
For generative models, one common approach is to generate outputs from the teacher and train the student on those outputs as targets.
This can simplify the target distribution and often makes the student easier to train, especially for seq2seq or instruction-following tasks.
Step-by-step / rationale distillation
For reasoning tasks, the student can be trained on the teacher’s intermediate reasoning traces as well as final answers.
This can transfer not only solution quality but decomposition style.
However, it also risks teaching the student to imitate surface rationales rather than underlying reasoning skill if the teacher traces are noisy or post-hoc.
Distillation design choices
Distillation can target different signals:
- final token distribution
- hidden states
- attention maps
- intermediate layer outputs
- final generated sequences
- verifier-approved reasoning traces
The more internal the target, the more complex the training pipeline. In practice, many production teams rely mainly on output-level or sequence-level distillation because it is easier to operationalize.
Quantization
Quantization reduces the number of bits used to represent weights, activations, or both.
Instead of storing tensors in FP32 or BF16, a quantized model may use lower-bit formats such as INT8, INT4, or other compressed representations.
Core idea
A floating-point tensor is mapped to a lower-precision representation using a scale and optionally a zero-point:
x_fp ~= scale * (x_int - zero_point)
Depending on the scheme, quantization may be:
- symmetric or asymmetric
- per-tensor, per-channel, or per-group
- static or dynamic
- weight-only or weight-plus-activation
Why quantization helps
Lower precision reduces:
- memory footprint
- memory bandwidth
- sometimes arithmetic cost
For many LLM inference workloads, memory bandwidth is the main bottleneck, so weight quantization can improve throughput even when compute is not the limiting factor.
Weight-only vs activation quantization
Weight-only quantization
Only the model weights are quantized. Activations remain in higher precision.
This is popular for LLM inference because:
- simpler to deploy
- often preserves quality reasonably well
- gives strong memory savings
Activation quantization
Activations are also quantized, which can unlock larger speedups but is harder because activations are more input-dependent and can have harder-to-handle outliers.
Training-aware methods or calibration are often required.
Outliers and granularity
A major challenge in low-bit LLM quantization is outlier sensitivity. A small number of unusually large values can dominate the dynamic range, forcing poor use of the available quantization levels.
This is why fine-grained scaling matters:
- per-channel quantization
- per-group quantization
- block-wise quantization
Granularity improves fidelity, but increases metadata and implementation complexity.
PTQ vs QAT
Post-training quantization (PTQ)
Quantize an already trained model, optionally with calibration data.
Advantages:
- cheap
- simple
- no retraining required
Disadvantages:
- quality loss can be significant at very low bit widths
Quantization-aware training (QAT)
Simulate quantization effects during training or fine-tuning so the model adapts to low-precision constraints.
Advantages:
- often better low-bit quality
Disadvantages:
- requires extra training
- more complex pipeline
Pruning
Pruning removes model parameters or structures judged low-importance.
Unlike quantization, pruning changes the model topology or effective sparsity pattern.
Unstructured pruning
Removes individual weights, producing sparse matrices.
Pros:
- can drastically reduce nonzero parameter count
Cons:
- often does not produce real speedups without specialized sparse kernels and hardware support
Structured pruning
Removes entire components such as:
- neurons
- channels
- attention heads
- MLP blocks
- layers or submodules
This is more actionable for runtime acceleration because the resulting model is structurally smaller.
Head pruning and layer dropping
In Transformers, not all attention heads contribute equally. Some pruning methods remove weak or redundant heads. Similarly, later fine-tuning can identify blocks or layers that can be reduced or skipped.
But pruning aggressiveness must be validated carefully. Apparent redundancy on one benchmark may hide real utility on another.
These methods solve different problems
A precise distinction:
- distillation transfers behavior into a smaller student architecture
- quantization compresses numeric representation while keeping topology mostly fixed
- pruning removes parts of the topology or induces sparsity
They are complementary, not mutually exclusive.
Example combined pipeline:
- distill large teacher -> smaller student
- apply PEFT or corrective fine-tuning
- quantize student for serving
- optionally prune structured components if supported by runtime stack
What they do not solve
These techniques do not eliminate all serving bottlenecks. For decoder-only LLMs, long-context inference is often heavily constrained by KV-cache growth. A quantized weight checkpoint may still be hard to serve efficiently if cache memory dominates runtime.
Therefore efficiency work must distinguish:
- static model weights
- dynamic inference-state memory
- runtime kernel efficiency
Why this matters in practice
These methods affect:
- model artifact size
- memory required per replica
- achievable batch size / concurrency
- latency and throughput
- hardware compatibility
- feasibility of edge or on-prem deployment
A team optimizing only for parameter count may miss the real bottleneck. In many deployment scenarios, bandwidth, cache growth, and kernel support matter at least as much as raw weight size.
Key takeaways
- Distillation creates a smaller student by training it to imitate a stronger teacher.
- Quantization reduces precision to shrink memory footprint and often improve throughput.
- Pruning removes parameters or structures, ideally in ways the runtime can exploit.
- PTQ is operationally simple; QAT usually preserves accuracy better at low bit widths.
- Structured pruning is usually more deployment-relevant than unstructured pruning.
- These methods are often most effective when combined rather than used in isolation.
Comments
0 comments
Article is closed for comments.