TL;DR: Context engineering is the practice of deciding what goes into a model’s context window, in what order, and at what cost, rather than only deciding how to phrase the request. It became the binding skill because the benchmark evidence turned decisively against the assumption that a large context window is a usable context window. The ICML 2025 NoLiMa benchmark tested thirteen models that all claim support for at least 128,000 tokens and found that eleven of them dropped below fifty percent of their own short-context baseline at just 32,000 tokens, with GPT-4o falling from 99.3 percent to 69.7 percent. IBM researchers measured function-calling degradation of 7 to 85 percent as tool catalogues grow, 7 to 91 percent as tool outputs lengthen, and 13 to 40 percent as conversations extend. The foundational TACL work on long contexts showed models retrieve well from the start and end of an input and poorly from the middle. Together these say the same thing: attention is a scarce resource that you allocate, not a free container that you fill. This guide gives you the measured degradation table, a derived advertised-versus-degradation ratio, a five-part context budget framework, and a practical audit. A CEO allocates scarce capital deliberately; a student re-measures when the next model ships.
For about three years, the story we told ourselves about working with language models was a story about phrasing. If the output was bad, the prompt was bad. Add a role. Add examples. Ask it to think step by step. Say “you are an expert.” A whole genre of advice grew up around the idea that the model contained the capability and the prompt was the key that unlocked it.
That story was never wrong exactly, but it has become a small part of a much larger problem, and the shift has a specific cause. Models got long context windows. Then people filled them. Then the results got worse in ways that no amount of rephrasing fixed, and the field had to develop a vocabulary for what was actually going on.
That vocabulary is context engineering, and the reason it matters is not that it is a fashionable term. It is that the failure it addresses is measured, published and large.
The definition, stated precisely
Prompt engineering asks: how do I phrase the instruction so the model does the right thing?
Context engineering asks a different and strictly larger question: what is the smallest set of high-signal information the model needs in its window in order to do this well, and how do I get exactly that in there and nothing else?
The distinction is easiest to see in what each one controls. Prompt engineering controls one component of the window, the instruction. Context engineering controls the whole window, which in any real system contains at least five competing occupants: the system instructions, the tool and function definitions, retrieved documents or knowledge, the conversation history, and the accumulated output of previous tool calls and actions.
That reframing matters because the instruction is usually the smallest of those five by an order of magnitude, and it is the only one most people ever touch. If your system is failing and you are still editing the wording of the prompt, you are optimising a rounding error.
Context engineering does not replace prompt engineering in the sense of making it obsolete. It subsumes it. Phrasing is one lever inside a larger allocation problem, in the same way that pricing is one lever inside running a business. Anyone who has worked through why prompt engineering alone is not enough will recognise the shape of this: the skill that looked like the whole job turns out to be one layer of a stack.
Why the window is not what it says on the label
Here is the evidence that forced the shift, and it is worth reading carefully because the numbers are more severe than the general discourse suggests.
The first result is foundational. Research published in the Transactions of the Association for Computational Linguistics on how language models use long contexts found a consistent U-shaped performance curve across multi-document question answering and key-value retrieval. Performance was highest when the relevant information appeared at the beginning or the end of the input, and degraded significantly when the model had to find it in the middle. Position within the window is not neutral. Where you put something changes whether the model can use it.
The second result is the one that should change your defaults. The NoLiMa benchmark, published at ICML 2025, was designed to close a loophole in the standard needle-in-a-haystack test. In the classic test, the hidden fact usually shares obvious wording with the question, so a model can succeed through literal matching without genuinely reasoning over the context. NoLiMa removed that crutch by constructing questions and target facts with minimal lexical overlap, forcing the model to infer the association.
The results across thirteen models that all advertise support for at least 128,000 tokens: performance was strong under 1,000 tokens, and eleven of the thirteen dropped below fifty percent of their own short-context baseline by 32,000 tokens. GPT-4o, one of the stronger performers, fell from 99.3 percent accuracy at baseline to 69.7 percent at extended length.
The third result moves this from retrieval into the agent workflows that most professional use now involves. IBM researchers introduced LongFuncEval to measure how well long-context models perform function calling as context grows, and reported three distinct degradation curves: a performance drop of 7 to 85 percent as the tool catalogue grows larger, 7 to 91 percent degradation in answer retrieval as tool responses get longer, and 13 to 40 percent degradation as multi-turn conversations extend.
Table 1: Measured long-context degradation (verified sources)
| Study | What it measured | Measured degradation |
|---|---|---|
| Lost in the Middle, Transactions of the ACL | Position of relevant information within the context | U-shaped curve: strongest at the start and end of input, significantly degraded in the middle |
| NoLiMa, ICML 2025 | Associative retrieval without lexical overlap, 13 models all claiming 128K or more | 11 of 13 models below 50 percent of their short-context baseline at 32K tokens; GPT-4o from 99.3 to 69.7 percent |
| LongFuncEval, IBM Research | Function calling as context grows | 7 to 85 percent drop as tool catalogue grows; 7 to 91 percent drop in answer retrieval as tool outputs lengthen; 13 to 40 percent drop as conversations extend |
Three independent research groups, three different task families, one consistent conclusion. Capacity to accept tokens and capacity to use them are separate properties, and the gap between them is where most real-world failure lives.
What the advertised numbers are actually worth
The marketing number and the working number are not the same, and it is useful to make the ratio explicit. The calculation below is a CEOtudent-derived ratio, not a measurement: it takes the one point at which NoLiMa observed most models falling below half their baseline and expresses it as a fraction of the window each model advertises.
Table 2: Advertised window versus observed halving point (CEOtudent editorial framework, derived from NoLiMa)
| Advertised context window | NoLiMa halving point observed across most tested models | Ratio of halving point to advertised window |
|---|---|---|
| 128,000 tokens | 32,000 tokens | 25.0 percent |
| 200,000 tokens | 32,000 tokens | 16.0 percent |
| 1,000,000 tokens | 32,000 tokens | 3.2 percent |
Read the method before you read the numbers. The 32,000-token figure is the point at which eleven of the thirteen NoLiMa models had fallen below half of their own short-context baseline on an associative-retrieval task designed to be hard. It is not a universal cliff, it does not transfer to every task type, and a model that fails this test at 32,000 tokens may still do useful literal-matching work far beyond it. What the ratio does establish is the direction and rough magnitude of the discount you should apply when a vendor quotes a window size. Treating an advertised million-token window as a million usable tokens is not a small error, it is an order-of-magnitude error on the hardest class of task.
The behavioural implication is simple and slightly counterintuitive: adding more context to a struggling system is usually the wrong move. Past a threshold that arrives much earlier than the label suggests, additional tokens dilute rather than inform.
The five-part context budget
If attention is scarce, then everything in the window is spending it. Here is the framework we use to audit that spend. Each row names an occupant of the window, the specific documented failure it drives, and the engineering move that addresses it.
Table 3: The context budget (CEOtudent editorial framework)
| Occupant of the window | Typical share of the problem | Documented failure it drives | The move |
|---|---|---|---|
| System instructions | Small in tokens, large in effect | Contradictory or stale rules that quietly override your request | Keep short, remove obsolete rules, state precedence explicitly |
| Tool and function definitions | Grows silently as you add capability | LongFuncEval: 7 to 85 percent degradation as the catalogue grows | Expose only the tools relevant to this task, not the full catalogue |
| Retrieved documents | Usually the largest occupant | Lost in the Middle: material buried mid-context is poorly retrieved | Retrieve less, rank harder, place the decisive material first or last |
| Conversation history | Grows monotonically until you intervene | LongFuncEval: 13 to 40 percent degradation as turns accumulate | Summarise and restart deliberately rather than letting threads run indefinitely |
| Accumulated tool output | The invisible one; grows fastest in agent workflows | LongFuncEval: 7 to 91 percent degradation as tool responses lengthen | Truncate and summarise tool returns before they re-enter the window |
The fifth row is where most people are losing without knowing it. In an agentic workflow, every tool call returns something, and that something stays in the window for every subsequent turn. A handful of verbose responses can consume more of the budget than the entire task description, and nothing in the interface tells you it happened. If you delegate multi-step work to AI systems, this is the single most valuable thing to instrument, and it pairs directly with the concepts covered in our guide to what to understand before delegating work to AI agents.
A practical audit you can run this week
You do not need infrastructure to start. You need a habit of asking what is in the window and whether it earns its place.
Question one: what is actually in there? Most people cannot answer this for their own workflows. List the five occupants for a task you run regularly and estimate the share of each. The estimate being rough does not matter; discovering that retrieved documents or tool history dominate is usually the entire insight.
Question two: what would I remove if the window were a tenth the size? This is the forcing function. It reliably identifies the material that is present out of habit rather than necessity. In most working setups, a large fraction of what is in the window is there because nobody removed it, not because someone decided it should be.
Question three: where is the decisive information sitting? Given the U-shaped curve documented in the TACL work, material buried in the middle of a long input is the least likely to be used. Move the thing the answer depends on to the front or the end, and check whether the output changes. It frequently does, and that is a cheap, self-run replication of a published effect.
Question four: is this failure a phrasing problem or a context problem? The diagnostic is straightforward. If the model produces a confident answer that is wrong about the facts you supplied, it is a context problem: the material was present but not used, or absent when you assumed it was there. If the model produces the right facts in the wrong form, it is a phrasing problem. The two look identical from the outside and have completely different fixes, and reaching for the prompt when the fault is in the context is the most common wasted hour in this work.
Question five: when did I last re-measure? Every number in this article is attached to a specific model generation. The NoLiMa results describe the models tested in 2025. The LongFuncEval curves describe the systems available when the paper was written. Each new generation moves these boundaries, generally in the right direction, and none of them has yet closed the gap between advertised and usable capacity. Building the habit of checking is more durable than memorising any particular threshold, which is the same argument we make about the skill of judging AI output.
Why this is a genuine skill and not a rebrand
There is a reasonable objection to all of this: that “context engineering” is prompt engineering with a more impressive name, invented because the old name started to sound unserious.
The objection fails on a specific test. The two disciplines are distinguished by what they can and cannot fix, and the boundary is sharp. No rephrasing of an instruction repairs a fact that was retrieved but placed in the dead middle of a 60,000-token input. No amount of role-playing or step-by-step framing recovers a function call that failed because forty tool definitions were loaded when four were relevant. Those are structural failures in the composition of the window, and they are only addressable by changing that composition.
That is the mark of a real discipline rather than a rebrand: it has its own failure modes, its own diagnostics, and its own fixes, none of which are reachable from the practice it grew out of.
The academic literature has already made the same call. A survey of context engineering for large language models, published in July 2025 after a systematic analysis of more than 1,400 research papers, treats it explicitly as a formal discipline that goes beyond prompt design, and decomposes it into three foundational components: context retrieval and generation, context processing, and context management. Those three map almost exactly onto the practical budget above, which is a reassuring sign that the framework is describing something real rather than something invented for a headline.
It is also worth being honest about who this is for. If your use of AI is a series of one-off conversational requests, prompt phrasing genuinely is most of your leverage and context engineering is largely academic. The moment you build anything persistent, meaning a workflow that retrieves documents, calls tools, or runs across many turns, the balance flips hard, and it flips without announcing itself. Understanding how language models actually work makes the reason obvious: the window is the model’s entire world for the duration of a request, and everything in it competes for the same finite attention.
The CEO and the student
The CEO framing is the one that makes this immediately intuitive. Context is a budget. It has a hard ceiling, the marginal value of what you put in it declines much faster than the ceiling suggests, and everything you add crowds out something else. Nobody would run a business by spending every available unit of capital simply because it was available. That is, precisely, what filling a context window looks like.
The student framing supplies the discipline the budget needs. Every threshold quoted here is a snapshot of a moving system. The right response to a published number is not to memorise it but to learn the test that produced it, so that when the next generation of models arrives you can find out where the new boundary sits instead of guessing. NoLiMa exists because someone noticed the standard test had a loophole and built a harder one. That instinct, checking whether the measurement actually measures what it claims, is the transferable part.
The people who will do best with these systems over the next few years will not be the ones with the best-phrased prompts. They will be the ones who know, at any moment, what is in the window and why.
Frequently asked questions
What is the difference between context engineering and prompt engineering in one sentence?
Prompt engineering optimises the instruction; context engineering optimises everything in the model’s window, of which the instruction is usually the smallest part.
Does a bigger context window solve the problem?
Not reliably. The NoLiMa benchmark tested thirteen models all claiming at least 128,000 tokens of support and found eleven of them below half their short-context baseline at 32,000 tokens. Capacity to accept tokens is not the same property as capacity to use them.
Is prompt engineering dead?
No. It is a component rather than the whole discipline. Phrasing still matters for single-turn conversational tasks, and it remains the right first diagnostic when the model produces correct facts in the wrong format.
What is the fastest improvement I can make?
Reduce what is in the window and move the decisive material to the start or the end. The U-shaped positional effect documented in the TACL work is easy to replicate on your own tasks in a few minutes, and the result usually surprises people.
Why do agent workflows degrade faster than chat?
Because they accumulate two occupants that chat does not: tool definitions and tool outputs. The IBM LongFuncEval measurements put degradation at 7 to 85 percent as the tool catalogue grows and 7 to 91 percent as tool responses lengthen, on top of the 13 to 40 percent that comes from conversation length alone.
Do I need special tooling to practise this?
No. The five audit questions in this article require nothing but attention to what you are sending. Tooling helps at scale, but the largest single gain, removing material that is in the window out of habit rather than necessity, is available to anyone immediately.
Sources
- Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics
- Modarressi, Deilamsalehy, Dernoncourt, Bui, Rossi, Yoon and Schütze, NoLiMa: Long-Context Evaluation Beyond Literal Matching, International Conference on Machine Learning, 2025
- Kate, Pedapati, Basu, Rizk, Chenthamarakshan, Chaudhury, Agarwal and Abdelaziz, LongFuncEval: Measuring the Effectiveness of Long Context Models for Function Calling, IBM Research, 2025
- Mei, Yao, Ge, Wang, Bi, Cai and colleagues, A Survey of Context Engineering for Large Language Models, July 2025
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
