TL;DR: “Keep a human in the loop” has become the standard answer to every question about AI agents, and it is not an answer. A loop has a shape: where the human sits, what they are shown, what they can stop, and what happens when they miss something. The public measurements let you design that shape instead of guessing. METR’s time-horizon work finds that the length of task an agent can complete at 80 percent success is roughly five times shorter than at 50 percent, with a 59-minute versus 15-minute gap for the strongest model in its sample, and that both horizons double at about the same rate, roughly every seven months. The tau-bench benchmark measured single-attempt success under 50 percent and eight-attempt consistency under 25 percent for state-of-the-art function-calling agents, a figure far above what random per-attempt failure would produce, which implies failures are concentrated in particular tasks. From those two findings plus Article 14 of the EU AI Act and the automation-bias literature, this piece builds a placement matrix keyed to two questions: how expensive is it to undo, and would you notice if it went wrong.
There is a specific moment that keeps repeating in professional work right now. An agent runs, produces a confident block of output, and a person looks at it for eleven seconds and approves it. Nothing in that moment is a checkpoint. It has the form of oversight and none of the function, and the person doing it usually knows.
The instinct to fix this by looking harder is wrong, because the problem is not attention. It is placement. A checkpoint in the wrong position costs the same attention and catches nothing.
What the reliability data actually says
Two measurements do most of the work here, and neither is about how smart the models are.
The first is METR’s time-horizon research, which asks a question with an unusually practical shape: how long a task, measured by how long a human expert takes to do it, can a model complete at a given success rate. Across models released between 2019 and 2025, the 50 percent time horizon has been growing exponentially with a doubling time of roughly seven months. That headline gets quoted constantly. The part that matters for checkpoint design almost never does.
METR also computed the horizon at 80 percent success. The doubling time is nearly identical, 213 days against 212 days for the 50 percent horizon, so the growth rate is not the issue. The levels are. In METR’s words, horizons at the higher reliability threshold are approximately five times shorter across the models studied, and the model with the longest 80 percent horizon in its sample, Claude 3.7 Sonnet, sat at around 15 minutes against a 50 percent horizon of 59 minutes, a gap of just under four times for that particular model.
| Measure (METR, models released 2019 to 2025) | 50 percent success rate | 80 percent success rate |
|---|---|---|
| Longest horizon measured in sample (Claude 3.7 Sonnet) | 59 minutes | about 15 minutes |
| Earlier frontier models (GPT-4, Claude 3 Opus) on the harder task suite | 5 to 30 minute range | Not separately reported |
| Doubling time of the horizon | 212 days | 213 days |
| Horizon at very high success rates, such as 95 percent | Not measured; METR states it cannot confidently measure these with its dataset | Not measured |
Verified data. Source: METR, Measuring AI Ability to Complete Long Tasks, section comparing 50 percent and 80 percent time horizons.
Read the last row carefully, because it is the most useful line in the table. Nobody has published a reliable measurement of how long a task an agent can complete at the reliability level you would actually want before letting something go unsupervised. The honest state of the evidence is that the high-reliability horizon is unmeasured, not that it is long.
The second measurement is about consistency rather than length. The tau-bench benchmark, built by Yao, Shinn, Razavi and Narasimhan, evaluates agents in simulated retail and airline domains where they must use tools, follow written domain policy, and converse with a simulated user. It introduced a metric called pass^k, which asks not whether an agent can succeed once but whether it succeeds on all k independent attempts at the same task. The paper’s finding is blunt: state-of-the-art function-calling agents such as gpt-4o succeed on under 50 percent of tasks, and pass^8 falls below 25 percent in the retail domain.
The inference nobody draws from pass^8
Here is where the two datasets stop being background and start being a design input.
If failures were random from attempt to attempt, an agent with a 50 percent chance on each try would succeed eight times in a row on about 0.4 percent of tasks. That is 0.5 raised to the eighth power. The measured figure is below 25 percent. Both benchmark figures are reported as upper bounds, so the exact multiple cannot be pinned down, but the measured consistency is more than an order of magnitude above what random per-attempt failure would produce.
| Model of failure | Predicted share of tasks passed all 8 times | Measured |
|---|---|---|
| Failures independent per attempt, uniform 50 percent chance | 0.4% | Not applicable |
| tau-bench retail, state-of-the-art function-calling agent | Not applicable | Under 25% |
CEOtudent editorial framework: the independence figure is calculated by CEOtudent from the benchmark’s reported single-attempt success rate, for comparison with the benchmark’s reported pass^8 result. The gap between the two is the finding, not either number alone.
The gap tells you the structure of the failures. If the observed eight-for-eight rate were near the independence prediction, agent errors would be essentially noise, and the correct response would be redundancy: run it twice, compare. It is not near the prediction. A large share of tasks succeed every single time and a large share fail every single time. The variation lives in the task, not in the run.
That single structural fact rules out the two most common supervision strategies in use today.
Random spot-checking is wrong, because the failures are not randomly distributed. Sampling one output in ten from a mixed pool will keep confirming the reliable tasks while the unreliable ones fail consistently, unwatched.
Running it twice and comparing is also wrong, for the same reason. A task the agent fails consistently will produce two confident, agreeing, wrong answers. Agreement between two runs is not evidence when the failure mode is task-specific.
What the structure calls for instead is classification. You have to know which tasks belong in the reliable pool and which do not, and the only way to know is to have measured, which is the discipline described in the delegation ceiling. The checkpoint goes on the class, not on the sample.
Where a checkpoint belongs: two questions, four answers
Everything above narrows to a placement rule. Two properties of a task determine where the human has to sit, and neither is about how hard the task is.
The first is reversibility: how expensive is it to undo. The second is detectability: if the agent got it wrong, would the error announce itself, or would it sit quietly in a spreadsheet for six weeks.
| Error is obvious | Error is silent | |
|---|---|---|
| Cheap to undo | Run free. No checkpoint. Fix on discovery, because discovery is guaranteed and the fix is cheap. Drafting, exploratory search, outlining, first-pass summaries. | Sample and score. Fixed-share review with a written scoring rule, on a defined task class rather than at random. Data extraction, tagging, categorisation, transcription. |
| Costly to undo | Gate before commit. One mandatory approval immediately before the irreversible step, not earlier in the chain. Publishing, sending, submitting, merging. | Pre-authorise and verify independently. Define the permitted action set before the run, and verify the result against a source that is not the agent’s own summary. Financial entries, external commitments, deletions, anything touching another party’s record. |
CEOtudent Checkpoint Placement Matrix. Editorial framework. The reversibility axis is standard decision-theory practice; the detectability axis and the four placements are CEOtudent’s construction.
Three things about this matrix are worth stating explicitly, because they are where people get it wrong.
The bottom-left cell says the gate goes immediately before the irreversible step, and that word immediately is load-bearing. A great deal of what passes for oversight is an approval placed at the start of a chain, on a plan, followed by unsupervised execution of that plan. Approving the plan is not a checkpoint on the outcome. The commit is the moment; the checkpoint belongs there.
The top-right cell is the one almost everyone under-serves. Silent errors in cheap-to-undo work feel harmless, which is exactly why they accumulate. A misclassified row costs nothing on the day and everything in the quarter when a decision is made on the aggregate.
The bottom-right cell contains the only instruction in the matrix that requires a second source. Asking the agent whether it did the task correctly is not verification; it is asking the failure to grade itself. The agent’s summary of its own run is generated by the same process that produced the run.
The oversight structure worth borrowing from regulation
There is a piece of drafting that solves a problem most individual workflows have not thought about, and it is sitting in law.
Article 14 of the European Union’s Artificial Intelligence Act sets out human oversight requirements for high-risk AI systems. It binds providers and deployers of those systems, not individuals using a chatbot, and this piece is not claiming otherwise. What makes it worth reading anyway is that it does not say “a human must review the output.” It specifies five distinct capabilities the overseeing person must be enabled to have, and those five map cleanly onto the failure modes of ordinary agent work.
| Article 14(4) requirement for high-risk AI systems | Translation to an individual agent workflow |
|---|---|
| (a) Properly understand the system’s capacities and limitations and monitor its operation, including detecting anomalies, dysfunctions and unexpected performance | Keep a written record of which task classes your agent fails on. Without that record you cannot classify, and without classification the matrix above is unusable. |
| (b) Remain aware of the possible tendency of automatically relying or over-relying on the output | Assume you will under-check. Make the check mechanical and scheduled rather than a matter of judgement in the moment. |
| (c) Correctly interpret the system’s output, taking into account available interpretation tools and methods | Require intermediate steps and sources, not conclusions. An answer you cannot trace is an answer you cannot check. |
| (d) Decide not to use the system, or to disregard, override or reverse the output | Keep a written list of actions the agent may never take unattended. A boundary that lives only in your head is not a boundary. |
| (e) Interrupt the system through a stop button or similar procedure allowing it to halt in a safe state | Define what “stopped safely” means before you start. Which state has to remain recoverable if you kill the run at minute four? |
Verified regulatory text in the left column, quoted in substance from Article 14(4) of Regulation 2024/1689. Right column is CEOtudent’s editorial translation and carries no legal status.
Requirement (e) is the one individual workflows systematically lack. Most people can stop an agent. Far fewer have decided in advance what state the work has to be in when they do. If your agent is halfway through renaming 400 files when you interrupt it, the question of whether that is recoverable was decided before you started, not when you hit the button.
The same structural instinct appears in the National Institute of Standards and Technology’s AI Risk Management Framework, which organises risk work into four functions, govern, map, measure and manage, rather than into a single review step. The common thread across both documents is that oversight is a system property, decided in advance, and not an act of attention performed at the end.
Why a checkpoint that exists can still fail
Placement is necessary and not sufficient, and the evidence for that predates the current generation of tools by more than a decade.
Goddard, Roudsari and Wyatt published a systematic review in the Journal of the American Medical Informatics Association in 2012 on automation bias, defined as the tendency to over-rely on automation. They screened 13,821 papers and included 74. The review is about clinical decision support, but its structural findings transfer directly, because the mechanism is about humans supervising machines rather than about medicine.
Two of its results should change how you build the checkpoint itself.
First, on what makes automation bias worse. The review identified environmental mediators including workload, task complexity and time constraint, which it describes as pressurising cognitive resources. This is the practical reason the eleven-second approval happens. A checkpoint scheduled into the busiest hour of your week is a checkpoint you have designed to fail, and no amount of intending to look carefully will fix a structural time constraint.
Second, on what mitigates it. Among the mitigators the review identified were implementation factors such as training and emphasising user accountability, and design factors including the position of advice on the screen, updated confidence levels attached to the output, and the provision of information versus recommendation.
That last item is the most actionable line in the whole review. Information versus recommendation. A system that hands you a recommendation invites approval; a system that hands you the evidence requires a decision. Translated into agent practice: instruct the agent to return what it found and what it did, with the intermediate steps, rather than a finished verdict for you to bless. The same prompt that produces a cleaner-looking output produces a weaker checkpoint. This is the operational core of briefing an agent properly, and it cuts against the instinct to ask for the tidiest possible answer.
The accountability finding is worth taking equally literally. Write your own name against the run before it starts, in the record, as the person answerable for the result. It sounds like theatre. The review found it works.
Putting it together: a worked pass
Take a concrete case. An agent is asked to go through 300 supplier invoices, extract amounts and dates, flag anomalies, and update a tracking sheet.
Classify the sub-tasks first, because the matrix operates on task classes and this is four tasks wearing one coat.
Extraction of amounts and dates is cheap to undo and silent when wrong: top-right. Sample and score, on a fixed share, with a written rule for what counts as correct. Not random across the whole batch; stratified by invoice format, because the tau-bench structure predicts that failures cluster by task type, and invoice format is the task type here.
Anomaly flagging is cheap to undo and obvious when wrong, at least in the false-positive direction: top-left. Run free. A false flag surfaces immediately when you look at it. The false-negative direction is silent, which is a real limitation, and the honest response is to treat the flagging as an assist and not as coverage.
Updating the tracking sheet is costly to undo and silent: bottom-right. Pre-authorise the permitted operations, append only, no overwrites, no deletions, and verify a sample of written rows against the source invoices rather than against the agent’s report of what it wrote.
Anything that triggers a payment is bottom-left and obvious enough: gate before commit, immediately before, with no batching of approvals.
Notice what this produces. Four different supervision regimes for one job, three of them cheaper than reading everything, one of them stricter than most people currently apply. That asymmetry is the point. Uniform review is simultaneously too expensive on the safe work and too weak on the dangerous work, which is precisely why it feels exhausting and still lets errors through.
Sizing the unattended block
One last practical consequence of the METR numbers.
If you want work to run unattended at something close to 80 percent reliability, the published horizon for the strongest measured model was about 15 minutes of equivalent human work. That is the unit to design around: segment the job so that no unattended stretch represents more than roughly a quarter of what the same system can attempt at coin-flip reliability.
Three honest caveats attach to that. METR’s tasks are heavily skewed towards software engineering and machine learning research, so the numbers do not transfer cleanly to invoice processing or email. The horizons are doubling roughly every seven months, so any specific minute figure has a short shelf life. And METR states plainly that it cannot confidently measure horizons at very high success rates, so the number you would most want is the number nobody has.
What survives all three caveats is the ratio and its direction. Higher demanded reliability buys dramatically shorter autonomous stretches, and the two horizons are improving at the same rate, which means the gap between them is not closing. Better models will let agents work longer. They will not, on this evidence, close the distance between “usually works” and “works reliably enough to leave alone.” That distance is where the checkpoint lives, and it is structural rather than temporary.
Which is the CEO and student pairing in its most literal form. The CEO decides in advance where the irreversible steps are and who answers for them. The student keeps the failure log current, because the classification is only as good as last month’s measurements. Neither half works without the other, and no amount of model improvement does either job for you.
Frequently asked questions
Is this only relevant if I use autonomous agents?
No. The matrix keys on reversibility and detectability, not autonomy level. A single chatbot answer you paste into a client email is a costly-to-undo, obvious-error task, which puts it in the gate-before-commit cell whether or not anything ran unattended. The distinction that matters is what the output touches, not how it was produced. The related boundary question is covered in which decisions to delegate and which to never automate.
Why is running the task twice and comparing not enough?
Because the tau-bench pass^8 result indicates failures cluster by task rather than scattering randomly across attempts. Two runs of a task the agent handles badly will tend to produce two similar, confident, wrong answers. Agreement between runs is evidence only when failures are independent, and the measured data says they are not.
Does Article 14 of the EU AI Act apply to me?
Almost certainly not in the way it applies to a provider of a high-risk system, and the piece does not suggest it does. It is used here as a well-drafted checklist of what oversight has to be able to do, because it names five specific capabilities rather than gesturing at review. Borrow the structure; do not claim the compliance.
How large a sample should I score in the sample-and-score cell?
The review literature does not give a universal number, and inventing one would be false precision. The design rule that follows from the data is about stratification rather than size: sample within task classes, not across the whole batch, because that is where the failures concentrate. Start with enough per class to distinguish a consistent failure from a one-off, then adjust from your own failure log.
What if my checkpoint keeps finding nothing?
Two possibilities, and they need different responses. Either that task class is genuinely reliable, in which case move it up the matrix and reclaim the time, or you are approving rather than checking. The automation-bias review points at the environment first: check whether the review is scheduled into a slot with real time in it, and whether the agent is handing you a recommendation to bless rather than evidence to judge.
Sources
Thomas Kwa, Ben West, Joel Becker and colleagues, Measuring AI Ability to Complete Long Tasks, METR, 2025, including the section comparing time horizons at 50 percent and 80 percent success rates.
Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024, introducing the pass^k reliability metric.
Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence, Article 14 on human oversight, in particular paragraphs 1, 2 and 4.
Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association, volume 19, issue 1, 2012.
National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework 1.0, 2023, and its four core functions.
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 2025, and the February 2026 note revising the organisation’s view of that result.
United States Census Bureau, Business Trends and Outlook Survey, artificial intelligence use estimates covering December 2025 to May 2026.
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
