Tokens, Context Windows, and Temperature
Make sense of the units models read, the context they can see, and the controls that change output variety.
Tokens are model units
A token is a piece of text chosen by a tokenizer. Common words may be one token, while unusual words can be split into several pieces. Punctuation, spaces, and numbers also affect the count. Tokenization differs by model and language, so a message with 1,000 characters does not always use the same number of tokens. Tokens matter because they influence context capacity, price, speed, and how much text a model can process in one request.
Why tokenization feels surprising
People think in words, but models operate on token sequences. A long technical term, a URL, code, or a language with different writing conventions may consume many tokens. Repeated text can still be expensive, even when it looks short on screen. When a request fails because it is too large, shorten repeated instructions, remove irrelevant history, or summarize older material. Always check the target model’s tokenizer when exact limits or costs matter.
const request = {
model: 'text-model',
input: 'Give three concise study tips.',
temperature: 0.2,
max_output_tokens: 60
};
console.log(JSON.stringify(request, null, 2));Context windows
A context window is the maximum amount of tokenized material available to one model request. It usually includes system instructions, the user prompt, conversation history, retrieved documents, tool results, and the requested answer. A larger window allows more background, but it does not guarantee equal attention to every detail. Relevant information can be buried, contradictory passages can confuse the model, and a very long prompt can increase latency and cost. Context is a budget, not unlimited memory.
Keeping context useful
Good context is selective, ordered, and clearly labeled. Put the current task and important constraints where the model can find them, remove duplicated history, and retrieve only evidence related to the question. For a long document, preserve headings and source references instead of pasting every page. Summaries save space but can lose exact wording, so retain original passages for high-stakes claims. Test the workflow with long, short, incomplete, and conflicting inputs.
Probability and choice
At each generation step, a model produces a probability distribution over possible next tokens. The highest-probability token is often sensible, but several alternatives may also be reasonable. A decoding method chooses among those candidates. Greedy decoding always selects the top option; sampling allows controlled variation. This is why the same prompt can produce different answers, particularly for creative tasks. Reproducibility depends on the model, settings, seed support, prompt, and surrounding system.
Temperature
Temperature changes the shape of the next-token probability distribution before a choice is made. Lower temperature concentrates probability around likely options, which often helps extraction, classification, and consistent formatting. Higher temperature spreads probability more widely, which can help brainstorming and creative variation but also increases surprising errors. Temperature does not add knowledge or intelligence. If a factual answer is wrong, lowering temperature may make the wrong answer more consistent rather than correct.
Other generation controls
Top-p sampling limits choices to the smallest group whose combined probability reaches a selected threshold. Top-k limits the number of candidates directly. Maximum output tokens caps response length, while stop sequences can end generation at a known marker. These settings interact, so change one at a time during testing. A short, explicit output schema is often more reliable than trying to repair an unconstrained response with decoding settings alone. Test these controls on real examples before choosing defaults.
Practical and responsible use
Estimate token needs before sending large files, set input and output limits, and show users clear errors when content is too long. Do not assume a bigger context window removes privacy concerns; every included passage may be processed or logged according to the service policy. For important workflows, evaluate accuracy, consistency, latency, and cost across representative examples. Keep critical instructions separate from untrusted documents, because extra context can also carry misleading or malicious instructions.