Context windows, tokens and temperature, explained

Three ideas explain most of the strange behaviour you get from an AI coding assistant: tokens, the context window, and temperature. None of them is complicated, and each one accounts for a different complaint.

Tokens

A model does not read characters or words. It reads tokens, which are chunks of text produced by splitting on statistical frequency. Common words are usually one token, rarer words split into several, and punctuation often stands alone.

For ordinary English prose the rule of thumb is about four characters per token, so roughly three quarters of a word each. That rule is a starting point, not a measurement, and it is worst exactly where developers use it:

• Code tokenises badly. Identifiers like getUserById split into pieces, and indentation, brackets and operators each cost tokens. A file of code uses noticeably more tokens per character than a paragraph of prose.
• JSON is expensive. Every quote, brace and comma is tokens spent on structure rather than content.
• Non-English text costs more, sometimes several times more per character.

The practical consequence is that "how much can I paste in" is a question about tokens, and your intuition from word counts will overestimate how much fits. If a number matters to you, count it with the tokenizer for the model you are using rather than estimating.

Tokens are also the billing unit, in both directions: what you send and what comes back. A long conversation resends its whole history with every message, so cost grows with the square of the conversation length rather than linearly.

The context window

The context window is the total number of tokens a model can consider at once. Everything competes for the same space:

• The system prompt
• The entire conversation so far
• Any files or documentation you pasted
• The response it is about to write

That last one matters and gets forgotten. The reply has to fit too, so filling the window with input leaves no room to answer.

When a conversation outgrows the window, something has to go, and what gets dropped is usually the oldest messages. This is why an assistant "forgets" the constraint you gave it twenty messages ago. It did not lose track. That text is no longer being sent.

The fix is not a longer conversation, it is a fresh one with the important context restated. A new chat that begins with the current file and the actual constraint will outperform a long thread where both scrolled out of range.

Lost in the middle

A bigger window does not mean the model uses all of it equally, and this is the least known of the three ideas.

The finding is from Lost in the Middle: How Language Models Use Long Contexts, which reports that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts".

The paper notes this holds even for models explicitly built for long contexts. So a million-token window is not a filing cabinet where everything is equally retrievable. It is more like attention that is sharpest at the edges.

This has a direct consequence for how you prompt:

• Put the important thing first or last. The file that matters, and the actual question, belong at the ends.
• Do not bury a constraint in the middle of a long paste and expect it to be honoured.
• Paste less. Three relevant files beat thirty, and not only for cost reasons. Padding the window with irrelevant material pushes what matters toward the middle.

"I gave it everything and it still got it wrong" is often this effect. Giving it everything is the cause rather than the remedy.

Temperature

Temperature controls how the next token is chosen from the model's ranked candidates.

At temperature 0 the model takes the highest-probability token every time, which makes it as close to deterministic as it gets: the same prompt tends to produce the same answer. Higher values flatten the distribution so less likely tokens get picked more often, producing more varied output.

The framing to avoid is that temperature is a creativity dial. It does not make the model more imaginative, it makes it less predictable. Those are different things, and for code the difference matters:

• Low temperature for code, extraction, and anything factual. You want the most likely answer, and you want to be able to reproduce it.
• Higher temperature for brainstorming, naming, or generating several options to choose between.

One caveat worth stating: temperature 0 does not guarantee identical output. Batching and floating-point behaviour on the provider's hardware can still produce small variations, so treat it as "much more consistent" rather than "reproducible".

Temperature also has nothing to do with accuracy. A low temperature produces the model's most likely answer, and the most likely answer can be confidently wrong. Turning it down makes hallucinations more consistent, not less frequent, which is why catching hallucinations is a separate job from tuning settings.

Structuring a prompt around all three

These three ideas point at the same practical advice, which is why they belong on one page rather than three.

Put the question at the end. If you paste a file and then ask, the question is at an edge where attention is strongest. If you ask and then paste two hundred lines, your question is now in the middle of the input.

Include only what is relevant. Every irrelevant file costs tokens, pushes the relevant material toward the middle, and gives the model more opportunities to answer about the wrong thing. Three files beat thirty.

State versions explicitly. "React 19, Next 16" costs about eight tokens and rules out a large slice of the version-drift problem, because it makes the correct answer more likely than the more commonly written one.

Start a new conversation when the topic changes. A fresh window with a restated problem consistently beats a long thread carrying twenty messages of unrelated history, which are still being resent and still competing for attention.

Turn temperature down for code. If the tool exposes it, low is right for anything where you want the most likely answer rather than a varied one.

None of this is prompt-engineering trickery. It is just arranging the input so the parts that matter sit where the model reads best, and leaving out the parts that do not.

What this explains

Put together, these three account for most of the friction:

"It was fine and then it got worse." The conversation outgrew the window and the early context was dropped. Start fresh with the relevant files.

"It ignored my instruction." The instruction was in the middle of a long prompt. Move it to the end.

"It contradicts itself between messages." Temperature above zero, plus a window that no longer holds the earlier answer.

"It works in the playground and not in my app." Different temperature, different system prompt, or a shorter effective window once your app's own prompt is included.

None of these is the model being unreliable in a mysterious way. They are consequences of a fixed-size input, an uneven attention pattern across it, and a sampling setting, and all three are things you can control.

Want to build the judgement that makes AI tools useful rather than confusing? Start with a track and learn enough to check the answers.