Purpose
This article explains how an LLM produces output text and introduces the generation parameters that most often appear in configuration files.
Overview
When an LLM generates text, it does not produce the entire answer at once. It produces one token at a time.
At each step, the model predicts probabilities over possible next tokens. A decoding strategy then chooses which token to output. That token is appended to the sequence, and the process repeats.
This continues until the model reaches a stopping point.
The generation loop
A simplified generation process looks like this:
- the model receives the prompt
- it computes probabilities for the next token
- a decoding strategy selects one token
- that token is appended to the text
- the model predicts again from the updated sequence
This loop continues until generation stops.
[Insert image here: step-by-step token generation diagram with repeated next-token prediction]
Greedy decoding
In greedy decoding, the model picks the most likely next token at each step.
This is simple and fast. It is often useful for baseline runs or deterministic behavior, but it can produce less diverse output.
Beam search
In beam search, the model keeps track of several candidate sequences instead of only one.
This is useful when you want to search more broadly for a strong output sequence. It is more computationally expensive than greedy decoding.
Common related parameters include:
generation_strategy: "beam_search"num_beams
Max new tokens
max_new_tokens sets the maximum number of tokens the model is allowed to generate.
This is one of the most common generation controls. It limits response length and helps prevent overly long outputs.
Why decoding matters
Two systems using the same model can behave quite differently if their decoding settings differ.
For example:
- a short
max_new_tokenslimit may truncate outputs - greedy decoding may produce narrow, predictable behavior
- beam search may explore more alternatives
That is why decoding parameters appear in both offline and online Giotto configurations.
Where these ideas appear in Giotto
You will see decoding parameters in places such as:
- offline
3_decoding - online
ray_serve_config.defaults.decoding - online per-agent decoding overrides
This is one of the clearest examples of how LLM theory maps directly to platform configuration.
Key takeaways
- LLMs generate text token by token.
- Decoding controls how the next token is chosen.
greedy,beam_search,num_beams, andmax_new_tokensare core runtime parameters.- These settings directly affect platform behavior.
Comments
0 comments
Article is closed for comments.