Context window

A context window is the maximum amount of text, counted in tokens, that a language model can read and write in one exchange. Tokens are pieces of words; in one clinical study, a 4,096-token limit worked out to roughly 2,400 words 1. The instructions, the record excerpt, any retrieved documents, and the model's answer all share the same window.

Why does the context window matter in healthcare?

The limit is shared between input and output. As one research team put it, "if a model has a context length of 2,048 and receives an input of 2,000 tokens, it can only generate 48 new tokens" 1.

Patient records fill windows fast. In a clinical decision-making benchmark built from 2,400 real hospital cases, the relevant information averaged 2,080 tokens per case and reached 15,023 tokens at the maximum 1.

When input exceeds the window, something has to give. The text is cut, summarized, or retrieved selectively. That study's framework asked the model to summarize what it had gathered whenever the conversation neared the limit 1. Each approach decides what the model never sees, so the design of retrieval-augmented generation deserves as much scrutiny as the choice of model.

Is a bigger window always better?

Bigger helps, within limits. In clinical summarization, performance generally improved with more in-context examples, and the number of examples a model could take depended on tokens per example and the window size. The best configuration used GPT-4 with a 32,000-token window 2.

Position matters, though. Studying long inputs, Liu and colleagues found that models used relevant information best when it sat at the beginning or end, and worst when it sat in the middle 3.

Quantity and order matter clinically too. In Hager and colleagues' study of open models on abdominal pathologies, the models often diagnosed best when given a single diagnostic exam alongside the history, rather than every exam. Reordering the same information changed accuracy by as much as 18%, and showing only abnormal laboratory results generally improved performance 1. Those models had 4,096-token windows. Newer models are larger, and the lesson to test for still holds.

Common misunderstandings

If it fits, the model reads all of it. Information in the middle of a long input is the most likely to be missed 3.

More context is always safer. In the clinical study, the models tested diagnosed worse when given every relevant exam 1. Curated context can beat complete context.

Only the user's text counts. Everything in the window can steer the output, including text inside retrieved documents or pasted notes. That is how prompt injection works.

What to ask a vendor

• What is the model's context window, and how much of it do your own instructions and retrieved material use?
• What happens when a chart exceeds it: truncation, summarization, or retrieval? Which parts get dropped first?
• Have you tested accuracy on long records, with key facts placed in the middle?
• Does the tool tell the clinician when input was cut?
• For an ambient scribe, how are long visits handled?

Related terms

• Large language model (LLM): the kind of model a context window belongs to.
• Fine-tuning: the alternative to fitting examples into the prompt.
• LLM evals vs clinical evaluation: why short benchmark prompts miss long-record failures.