TL;DR: Computer-use agents, the systems that see your screen and drive your mouse and keyboard, made a genuine capability jump, and the demos are real. But the published evidence draws a sharp line that the demos do not. On OSWorld, the academic benchmark of 369 short real-desktop tasks, agents went from 12.24 percent success at launch in 2024 to scores above the 72.36 percent human baseline. On OSWorld 2.0, released in June 2026 with 108 long-horizon workflows that take a human a median of 1.6 hours, the best frontier agent completes 20.6 percent. The same top system scores 83.5 percent on the short benchmark and 20.6 percent on the long one. That gap is the single most important number in this field, and it tells you exactly where delegation works and where it destroys value. This guide gives you the verified benchmark table, a derived cost-per-finished-task analysis that shows what unreliability really prices at, the published six-mode failure taxonomy, and a three-tier delegation framework that maps office tasks to the supervision they actually need. A CEO buys reliability, not demos; a student re-runs the check when the next model ships.
Something genuinely changed in the last two years. An AI system can now look at your screen, find the button, click it, type into the field, switch applications, and keep going. Not through a bespoke API integration built by your IT department, but the same way you do it, by looking and acting. This is the capability people have been imagining since the first chatbot, because it is the one that finally touches the actual shape of office work, which is not a conversation but a long sequence of small manipulations across half a dozen applications that never talk to each other properly.
The demo videos are compelling and, importantly, they are not fake. The problem is that a demo is a sample of one from a distribution nobody shows you. The useful question is not whether an agent can book a flight or reconcile a spreadsheet, because the answer to that is yes, sometimes. The useful question is: how often, on what kind of task, and what happens the rest of the time. That question has real answers now, published in peer-reviewed and preprint benchmark research, and the answers are far more actionable than either the hype or the backlash.
The benchmark that defines the field, and the one that redefined it
OSWorld, introduced at NeurIPS 2024, was the first serious attempt to measure this properly. It puts an agent in a real operating system with real applications and gives it 369 tasks drawn from actual computer use: file management, web browsing, document editing, and workflows that span multiple applications. Each task has a scripted, execution-based check, so success means the file really ended up in the right state, not that a judge model liked the transcript.
Two numbers from that original work still anchor everything. Humans complete 72.36 percent of the tasks, which is itself worth sitting with, because it means these are not trivial errands even for a person. And the best model at launch in April 2024 completed 12.24 percent. The authors attributed the gap mostly to GUI grounding, knowing where on the screen the thing actually is, and operational knowledge, knowing the conventional way an application expects to be driven.
That gap closed fast. Over roughly two years, reported OSWorld scores climbed past the human baseline, and by mid-2026 leading systems report figures in the eighties. Read in isolation, that is a story of solved-and-shipped, and a lot of 2026 commentary read it exactly that way.
Then, in June 2026, the same research group published OSWorld 2.0, and the picture changed. The new benchmark contains 108 workflows, each one a realistic end-to-end job rather than a single errand. The design constraint is the interesting part: each task takes a human user a median of about 1.6 hours. Where a task in the original benchmark took an agent roughly 30 tool calls, tasks in the new one average 318.4 calls for a frontier model running with maximum reasoning effort. This is not a harder version of the same test. It is a different regime, and it is much closer to what a real job looks like.
On that benchmark, at a 500-step budget, the best frontier agent completes 20.6 percent of tasks.
Reading the results table properly
The following table reports the published OSWorld 2.0 results at a 500-step budget with maximum reasoning effort, as documented by the benchmark authors. Binary completion means the whole workflow finished correctly. Partial score credits progress along the way. Cost is the approximate reported spend per task attempt.
| System (as reported) | Binary completion | Partial score | Approx. cost per attempt |
|---|---|---|---|
| Claude Opus 4.8 (batched) | 20.6% | 54.8% | $72.4 |
| Claude Opus 4.8 (single) | 18.5% | 49.3% | $76.1 |
| Claude Opus 4.7 (batched) | 18.2% | 48.9% | $33.6 |
| Claude Opus 4.7 (single) | 13.9% | 49.1% | $35.8 |
| GPT-5.5 (batched) | 13.0% | 49.5% | $25.5 |
| Claude Sonnet 4.6 | 8.3% | 41.5% | $22.3 |
| MiniMax M3 | 4.6% | 22.3% | $2.4 |
| Kimi 2.6 | 4.6% | 22.1% | $6.6 |
| Qwen 3.7-Plus | 2.8% | 21.5% | $3.8 |
Two features of this table matter more than the ranking.
The first is the distance between the binary and partial columns. The leading system finishes one task in five outright, but scores 54.8 percent on partial progress. Agents are not failing at the starting line. They are getting substantially through long workflows and then not landing them. For a manager deciding what to delegate, this is the entire story: the output of a computer-use agent on a long task is usually a half-finished job, not a refusal. Half-finished work that looks finished is the most expensive kind of output there is, because someone has to audit it before they can trust it, and auditing a workflow you did not perform is often slower than performing it.
The second is the benchmark cliff. The authors note that the same Claude Opus 4.8 configuration that reaches only 20.6 percent binary completion on the long benchmark scores 83.5 percent on the original OSWorld. One system, one moment in time, two numbers four times apart. Any claim about agent reliability that does not specify task length is close to meaningless.
What unreliability actually costs
The published table gives cost per attempt, but that is not the number a business cares about. What you want to know is the cost per finished task, which means dividing the spend by the probability the attempt actually lands. The following table, a CEOtudent editorial framework applied to the published figures above, derives that.
| System (as reported) | Binary completion | Cost per attempt | Derived cost per finished task |
|---|---|---|---|
| MiniMax M3 | 4.6% | $2.4 | ~$52 |
| Qwen 3.7-Plus | 2.8% | $3.8 | ~$136 |
| Kimi 2.6 | 4.6% | $6.6 | ~$143 |
| Claude Opus 4.7 (batched) | 18.2% | $33.6 | ~$185 |
| GPT-5.5 (batched) | 13.0% | $25.5 | ~$196 |
| Claude Opus 4.7 (single) | 13.9% | $35.8 | ~$258 |
| Claude Sonnet 4.6 | 8.3% | $22.3 | ~$269 |
| Claude Opus 4.8 (batched) | 20.6% | $72.4 | ~$352 |
| Claude Opus 4.8 (single) | 18.5% | $76.1 | ~$411 |
Read this as an illustration of what unreliability prices at, not as a quote. The arithmetic assumes retries are independent and that a failed attempt is worth nothing, and neither is strictly true: a failed run often leaves usable partial work, and a task the agent fails once it will often fail again for the same structural reason rather than succeeding on a fresh roll. Treat these as an upper bound on the naive cost of brute-force retrying.
Even as an upper bound, the shape is instructive. The cheapest system per finished task is also the one that finishes almost nothing, which means low cost per success is not a buying signal when the base rate is 4.6 percent; you would be waiting through twenty-one attempts. And the most capable system is the most expensive per success by a wide margin, because you pay frontier prices on every failed attempt too.
Now put a human next to it. The benchmark tells us these workflows take a person a median of 1.6 hours. At a fully-loaded knowledge-worker cost of somewhere between $30 and $80 an hour, a figure that varies enormously by role and market and which we offer purely as an illustrative range, the human does the job for roughly $48 to $128, at a completion rate near certainty and with the judgment to flag when the task itself was wrongly specified. The agent’s raw compute for one attempt is competitive with that. Its cost per actually-finished long workflow is not, yet.
This is not an argument against the technology. It is an argument about which tasks to point it at, which is a different and much more useful conclusion.
Why long tasks break: the published failure taxonomy
The OSWorld 2.0 authors did the genuinely valuable work of categorizing how agents fail rather than just counting that they do. Six patterns recur:
Information tracking. Agents miss information that was present in the instruction, visible in the environment, or supplied through the user channel.
Timing sensitivity. On time-sensitive steps, long gaps between observing and acting mean the action lands on an interface state that has already changed.
Domain artifacts. Agents struggle to correctly interpret and produce domain-specific file formats and conventions.
Verification gaps. Agents frequently fail to check task-critical properties before submitting, and, strikingly, fail to correct errors they had already noticed.
State drift. Information gathered early is forgotten when the task state exists only inside compressed reasoning context rather than in the environment.
Hidden state. Every system tested scores weakest where success requires inferring something never stated: unstated requirements, tracking several items at once, resolving conflicting instructions, and adapting to environments that change underneath them.
Underneath all six sits one finding that deserves to be the headline. The authors report that every model spends almost none of its budget detecting and fixing its own mistakes, with recovery and repair together accounting for below 7 percent of effort across all systems.
That is the mechanism. These systems are not weak at doing things. They are weak at noticing that what they just did was wrong. A competent human doing a 1.6-hour workflow spends a considerable share of that time checking their own work, and reflexively re-checks after anything surprising. An agent that spends under 7 percent of its budget on self-correction will carry a small early error all the way to the end and hand you a confidently completed, wrong result. This is precisely the failure profile that the delegation ceiling describes: the constraint is not capability, it is the verification burden that capability transfers back to you.
The delegation tier framework
Put the evidence together and a practical rule emerges. Reliability is a function of task length, application count, and whether the outcome is checkable in less time than it took to produce. The following three-tier framework, a CEOtudent editorial framework built on the failure patterns above, maps office work to the supervision it actually needs.
| Tier | Task profile | Evidence basis | How to run it | Realistic 2026 posture |
|---|---|---|---|---|
| Tier 1: Delegate | Short, single-application, one clear end state, output verifiable at a glance (rename and file a batch, extract fields from a form, reformat a document, pull a defined report) | Short-task benchmark performance now exceeds the 72.36% human baseline | Fire and spot-check. Sample the output rather than auditing every item | Genuine time savings available today |
| Tier 2: Supervise | Multi-step, two or three applications, 15 to 45 minutes of human time, reversible if wrong (draft a competitive summary from named sources, prepare a recurring deck, reconcile two lists) | Partial scores near 50% mean substantial progress, low binary completion means poor landing | Let the agent produce the draft, then verify at named checkpoints. Never accept the final state unread | The realistic sweet spot; you are buying a first draft, not an outcome |
| Tier 3: Do not delegate yet | Long-horizon, four or more applications, over an hour of human time, or any irreversible action (sending, paying, publishing, deleting, submitting) | 20.6% binary completion on 1.6-hour workflows; under 7% of budget spent on self-correction | Decompose into Tier 1 and Tier 2 units with human checkpoints between them, or do it yourself | Unattended execution is not supported by the evidence |
The decomposition move in Tier 3 is the one that pays. A workflow that fails as a single 318-call instruction often succeeds as five supervised segments, because you are inserting the verification the agent does not perform for itself. You are not working around a limitation so much as supplying the missing component.
The irreversibility rule
One line cuts across all three tiers and is worth stating on its own, because it is the rule that prevents the expensive kind of mistake.
Never grant unsupervised authority over an action that cannot be undone. Sending an email, submitting a form, making a payment, publishing a post, deleting a record, accepting terms. The benchmark evidence gives an unusually crisp justification: a system that spends under 7 percent of its effort on self-correction and scores 20.6 percent on long workflows will, on most attempts, arrive at the irreversible step in a state it has not verified. The failure mode is not that it refuses. It is that it proceeds confidently.
The cost asymmetry does the rest of the argument. A reversible error costs you the time to redo it. An irreversible one costs whatever it costs, and you find out afterwards. Requiring a human confirmation at exactly the irreversible steps costs seconds and eliminates the entire tail. This is standard operational risk management, and it applies here unchanged.
How to evaluate the next model yourself
The specific numbers in this article will age, and that is fine, because the method will not. When the next agent launches with an impressive figure attached, four questions separate a real capability claim from a demo.
Which benchmark, and how long are the tasks? A score on short single-app tasks tells you nothing about hour-long multi-app work. Ask for the long-horizon number specifically. The four-times gap between the two OSWorld generations is the reason.
Binary completion or partial credit? Partial scores near 50 percent alongside binary scores near 20 percent describe a system that gets most of the way and does not land. Those are different products. Insist on the binary number.
What was the step and cost budget? Results at 500 steps and maximum reasoning effort are not the configuration you get by default, and the cost per attempt at that setting can be substantial.
What is the self-correction rate? This is the leading indicator. When published recovery-and-repair effort moves meaningfully above the current sub-7-percent range, the reliability picture will change materially, and Tier 3 will start to open up. Until then, treat unattended long-horizon execution as unsupported regardless of the headline score.
This is the same evaluation discipline that applies to choosing a model for any task: read the benchmark that matches your actual job, not the one that flatters the launch.
The CEO and the student
A CEO does not buy a capability because a demonstration was impressive. They ask what it reliably delivers, at what cost, with what failure mode, and what the failure costs when it happens. On that standard, computer-use agents in 2026 are a clear buy for short bounded tasks, a strong buy for supervised drafting of medium workflows, and a clear hold for unattended long-horizon execution. That is not a hedge. It is three distinct decisions, and the evidence supports each of them separately.
The student half is what keeps this from calcifying. This field moved from 12.24 percent to above the human baseline on short tasks in about two years. The long-horizon number will move too. The person who wrote off agents in 2024 because of a bad demo is now wrong, and the person who declares long-horizon delegation solved today will be wrong in the other direction. What survives both errors is the habit of going back to the benchmark, asking the four questions above, and moving a task up a tier only when the evidence says you can.
Build the tier map for your own work now. Most people find they have more Tier 1 tasks than they expected, which is where the immediate return is, and that a few of their Tier 3 workflows decompose cleanly into supervised segments. That is the whole opportunity in 2026: not the autonomous office, but a considerably faster one, run by someone who knows exactly where to look.
Frequently asked questions
Are agentic browsers different from computer-use agents?
An agentic browser operates inside the browser, so its world is web pages, tabs, and forms. A computer-use agent drives the whole desktop, including local applications and the file system. Browser-scoped agents tend to be more reliable within their narrower domain, since the environment is more uniform, but most real office workflows cross the boundary between browser and desktop, which is exactly where multi-application failure modes appear.
If agents beat the human baseline on OSWorld, why not delegate everything?
Because that baseline applies to short, mostly single-application tasks. The 72.36 percent human figure and the agent scores above it both come from the 369-task benchmark. On the long-horizon benchmark, where tasks take a human a median of 1.6 hours, the best agent completes 20.6 percent. Task length is the variable that decides delegation, not the headline score.
Is partial completion useful, or is it worthless?
It depends entirely on whether you planned to check the work. If you expected a finished outcome, partial completion is worse than worthless, because it looks finished and quietly transfers an auditing job to you. If you asked for a first draft and intended to review it, a partial score near 50 percent is real value. This is why Tier 2 exists as a separate category.
What is the single best predictor that reliability is improving?
The share of effort a system spends detecting and correcting its own errors. Published figures currently put recovery and repair below 7 percent across all systems tested. Because most long-horizon failures trace back to uncorrected early mistakes, that number rising is the signal that unattended execution is becoming viable.
Should small teams wait for the technology to mature?
No. Tier 1 delegation works now and is where the immediate return sits. What small teams should not do is build a process that depends on unattended long-horizon execution, because the current evidence does not support it, and a process built on a capability that does not exist yet fails quietly and expensively.
Sources
- Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, NeurIPS 2024 Datasets and Benchmarks Track (arXiv:2404.07972)
- Yuan et al., OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks, June 2026 (arXiv:2606.29537)
- XLANG Lab, University of Hong Kong, OSWorld benchmark documentation and verified results methodology
- OECD, Artificial Intelligence and the Future of Work, policy analysis on task-level automation and complementarity
- Stanford Institute for Human-Centered Artificial Intelligence, AI Index Report, chapters on technical performance and benchmark saturation
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
