Purpose
This article explains how LLMs generate text at inference time, how decoding strategies change model behavior, and how search-based methods extend simple next-token generation.
Overview
At inference time, an autoregressive LLM performs an iterative decode loop.
For a current prefix x_<t, the model computes logits over the vocabulary for the next token x_t, transforms those logits into a selection distribution, chooses one token according to a decoding rule, appends it to the sequence, and repeats.
This process continues until one of the following occurs:
- EOS token is generated
- a stop sequence is reached
- max token budget is exhausted
- an application-level termination rule fires
Although the model weights are fixed, output behavior depends strongly on the decoding procedure. In practice, inference quality is a function of both:
- model quality
- decoding and search policy
Logits, probabilities, and temperature
The model outputs a vector of logits z in R^|V|, one score per vocabulary token.
Probabilities are obtained via softmax:
p_i = exp(z_i / T) / sum_j exp(z_j / T)
where T is temperature.
Temperature effects
T < 1: sharpens the distribution, making high-probability tokens even more dominantT = 1: unchanged distributionT > 1: flattens the distribution, increasing diversity
Temperature is therefore a control over entropy, not a direct “creativity” knob in a vague sense.
At T -> 0, decoding approaches argmax selection.
Greedy decoding
Greedy decoding selects:
x_t = argmax_i p_i
at every step.
Advantages:
- deterministic
- cheap
- easy to debug
Disadvantages:
- myopic
- prone to repetition or local optima
- may lock into a bad trajectory early
For reasoning-heavy or open-ended tasks, greedy decoding often underexplores the solution space.
Sampling
Sampling draws the next token from the model distribution rather than always taking argmax.
This allows lower-probability but still plausible continuations to be explored.
Common sampling controls
Top-k sampling
Restrict sampling to the top k tokens by probability, then renormalize.
Top-p / nucleus sampling
Choose the smallest token set whose cumulative probability exceeds threshold p, then sample from that set.
Top-p adapts to the shape of the distribution and is often preferred over fixed top-k.
Min-p and other truncation rules
Additional heuristics can suppress very low-probability tails or normalize exploration more tightly.
Tradeoff
Sampling improves diversity and can help reasoning when multiple plausible paths exist, but it increases variance and can produce instability if parameters are not well tuned.
Beam search
Beam search maintains multiple partial hypotheses simultaneously.
At each decode step:
- expand each beam with candidate next tokens
- score resulting partial sequences
- keep only the top
Bbeams
where B is beam width.
Why beam search helps
Beam search is less myopic than greedy decoding because it keeps several hypotheses alive. This is useful for:
- machine translation
- structured generation
- tasks where sequence likelihood is a good proxy for output quality
Limitation for open-ended text
For open-ended dialogue or creative generation, beam search often produces bland or repetitive outputs because maximizing sequence likelihood is not always aligned with human preference.
Length effects and normalization
Sequence scores are products of token probabilities, so longer sequences tend to accumulate lower raw probability.
Beam search systems therefore often use:
- length normalization
- coverage penalties
- repetition penalties
Without such adjustments, decoding may be biased toward unnaturally short outputs.
Repetition controls and logits processing
Production decoding stacks often modify logits before token selection using heuristics such as:
- repetition penalty
- no-repeat n-gram constraints
- bad-word masking
- forced BOS/EOS or schema tokens
- stop-sequence handling
- tool-call grammar constraints
These are not part of the base model; they are inference-time controls layered on top of the logits.
This is important operationally because visible output behavior can change significantly even when the model checkpoint is unchanged.
KV-cache and incremental decoding
Autoregressive generation would be too expensive if every step recomputed attention over the full prefix from scratch.
Instead, decoder-only systems use a KV cache:
- store keys and values for past tokens at each layer
- compute only new states for the newest token
- reuse previous cache entries
This reduces repeated computation dramatically and is fundamental to practical serving.
Prefill vs decode
Inference is usually split into two phases:
Prefill
Process the full prompt in parallel to initialize hidden states and KV cache.
Decode
Generate new tokens one by one, reusing cache.
This distinction matters because:
- prefill is throughput-heavy and parallel
- decode is latency-sensitive and sequential
Serving stacks optimize these phases differently.
Multi-token prediction and speculative decoding
Multi-token prediction
Some frontier directions train models to predict more than one future token per step. This changes both supervision structure and possible inference designs.
Speculative decoding
A smaller draft model proposes several future tokens, and a larger target model verifies them. If accepted, multiple tokens are committed at once.
This can reduce latency by amortizing expensive target-model computation.
Speculative decoding is an inference-system optimization, not a change in the base Transformer block, but it can materially improve serving efficiency.
Test-time search
Decoding does not need to follow a single token path. For hard tasks, systems can perform explicit search over candidate completions.
Examples:
- Best-of-N
- beam search
- self-consistency
- verifier-guided sampling
- MCTS-like rollout strategies
- self-refinement loops
These methods treat inference as a search problem over possible solutions rather than a single stochastic decode.
Best-of-N
Generate N complete candidates, then pick the best according to:
- reward model
- verifier
- heuristic score
- majority answer agreement
- external evaluator
This is simple and often very effective, especially when:
- sample diversity is meaningful
- verification is cheaper than generation error
Its main drawback is cost: generation expense scales roughly with N.
Verifier-guided inference
For domains like code and math, inference can be coupled to a verifier:
- run tests
- check formal constraints
- validate output schema
- reject invalid candidates
This changes the system from “generate once” to “generate, check, keep or revise.”
Such setups often outperform plain decoding on correctness-critical tasks.
Tree and rollout-based search
More advanced systems use tree search:
- partial hypotheses as nodes
- rollout or continuation sampling
- heuristic or reward-based node evaluation
- pruning under a search budget
This includes Tree-of-Thoughts-like inference and MCTS-inspired reasoning pipelines.
These methods are expensive, but they can be valuable when the solution space is highly branching and local decoding decisions are brittle.
Why this matters in practice
Two deployments using the same model can behave very differently because of differences in:
- temperature
- sampling policy
- repetition penalties
- beam width
- stop conditions
- verifier use
- test-time search budget
This is why model evaluation must record inference configuration. “Model performance” without decode settings is often underspecified.
Operationally, inference design affects:
- latency
- throughput
- determinism
- answer diversity
- reasoning success rate
- infrastructure cost
Key takeaways
- Inference is an iterative autoregressive loop from logits to token choices.
- Temperature rescales logits and changes output entropy.
- Greedy, sampling, and beam search explore different quality-diversity tradeoffs.
- KV-cache reuse is fundamental to practical LLM serving.
- Test-time search can substantially improve hard-task performance by exploring multiple candidate trajectories.
- Decoding configuration is often as important as the base checkpoint when evaluating real system behavior.
Comments
0 comments
Article is closed for comments.