AI Benchmarks Miss 90% of US Working Hours: What It Means

A research post from Mercor, a company that builds tests and data for AI models, was being widely discussed on Hacker News when we saw it on 15 and 16 October 2026. Its headline estimate is that about 90 percent of American working hours¹ go to tasks with little or no coverage from existing AI benchmarks. ¹This piece answers that news item. It explains what AI benchmarks are, how the number was built, what it does not say about automation and jobs, and what the story of the bank teller suggests. As of 17 October 2026 the post calls its own result provisional, and nothing here is advice about careers, studies or investing.

What are AI benchmarks, and what did Mercor measure?

An AI benchmark is a fixed set of test tasks with a scoring rule, so that different models can be compared on the same work. Mercor's AI benchmark analysis (https://www.mercor.com/research/labor-index/benchmark-map/), titled "A benchmark map of the US economy", asks how much of the economy those tests actually touch. It starts from O*NET, a public catalogue that lists the tasks inside each occupation: 923 occupations in 22 job families, with 18,797 tasks¹ in all. ¹We opened the post and read its method section in full.

Mercor then chose 29 benchmarks using five criteria: at least 25 public tasks, tasks where a model must act rather than only answer, realistic enterprise work, a spread across job families, and tests that frontier models have not yet saturated. A language-model judge tagged each benchmark prompt with the O*NET tasks it appears to match, following the method of Wang and colleagues. A task counts as covered when it matches at least five prompts. On that rule only 208 of the 923 occupations have a covered task, and six of the 22 families have none. ¹

To move from tasks to hours, Mercor combined those tags with an estimate of how much working time each task takes. That estimate comes from other researchers, Hatgis-Kessell and colleagues, who derived it with a language model. The result is the headline: about 90 percent of working hours sit on tasks with little or no benchmark coverage. ¹

Which work do AI benchmarks leave out?

The gaps are not random. Using a separate classification of which tasks AI could plausibly help with, Mercor found that exposed tasks are about six times more likely to be covered than unexposed ones. Tasks in occupations with a large total wage bill are also over-represented. Lawyers' tasks are well covered, while legal secretaries and court clerks are barely covered, so the post describes a long tail of smaller, middle-income knowledge jobs as undercovered. It also cites an estimate that only 37 percent of US jobs can be done entirely from home, which leaves a large share of work with physical parts that current tests do not capture. ¹

An academic team arrived at a similar picture by another route. In a preprint on arXiv, Zora Zhiruo Wang and nine co-authors analysed 43 agent benchmarks containing 72,342 tasks², mapped them to 1,016 US occupations, and found that agent development is heavily focused on programming while employment and economic value sit elsewhere. ²Computer and math work accounts for only 7.6 percent of US employment, yet it dominates the tests. Management, legal, and architecture and engineering make up just 1.4, 0.3 and 0.7 percent of the 19,179 examples the paper analysed for that comparison, even though 88, 70 and 71 percent of that work is digital. The paper (https://arxiv.org/abs/2603.01203) is a preprint, so it may not have been through peer review. ²

Why would AI benchmarks cluster in a few jobs?

Tests are easiest to build where answers can be checked automatically and where the work already happens on a computer. Code either passes its tests or it does not. A conversation with a grieving relative, or a repair on a pipe, is much harder to turn into a scoring rule. That is a plausible reason for the skew, though neither source tests it directly, so we treat it as our reading rather than a finding.

Does the 90 percent mean AI cannot do most work?

The number measures whether AI benchmarks exist, not whether AI can do the work. A task with no benchmark might be easy for a model or impossible for one; the figure simply says there is no standard way to know. Reading it as "90 percent of work is safe" or "90 percent of work is at risk" would both be mistakes.

Mercor lists its own limits. The 29-benchmark list is not a complete or representative sample, and an incomplete list will bias coverage estimates downward. Per-family caps understate coverage in heavily tested fields such as software engineering. Choices such as the five-prompt threshold, the language-model tagging and the language-model hours weights all move the result. ¹The authors say they validated the tags against human expert tags on a subset of one benchmark family, not across all 29.

There is also a business context. Mercor describes its own offer as human experts who supply data and evaluations to AI developers, and it runs a family of work-realism benchmarks called APEX. ³A finding that the world needs more and better evaluations fits that business. That does not make the numbers wrong, and the Carnegie Mellon and Stanford paper points the same way, but it is a reason to read the 90 percent as an estimate by an interested party.

What does history say about automation and jobs?

The best-known case is the bank teller. Economist James Bessen wrote in 2015 (https://www.imf.org/external/pubs/ft/fandd/2015/03/bessen.htm) that cash machines automated routine tasks such as dispensing cash and taking deposits. In the average urban market, the tellers needed to run a branch fell from 20 to 13 between 1988 and 2004⁴, yet urban branches rose 43 percent, because cheaper branches were worth opening, and the remaining teller work shifted toward selling and relationships. Teller jobs did not decline as cash machines spread. ⁴The US Bureau of Labor Statistics (https://www.bls.gov/opub/btn/volume-7/in-the-money-occupational-projections-for-the-financial-industry.htm) adds that teller employment grew 11.6 percent between 1997 and 2007. ⁵

The story does not end there. The same BLS article, from October 2018, projected teller employment to fall 8.3 percent between 2016 and 2026, from 502.7 thousand to 460.9 thousand, as online and mobile banking let customers skip the branch. ⁵So the lesson is not that technology never destroys jobs. It is that automating a task and losing a job are different events, and a different technology, online banking, is the one BLS cites for the later decline.

The same logic is why pay tends to follow output per worker over long periods, as we explain in why wages rise (https://therealigned.com/story/why-wages-rise), and why changes in how scarce labour is can reshape pay, as in the Black Death and wages (https://therealigned.com/story/black-death-and-wages). Tools that change which tasks people do, and what a worker's hour is worth, have always worked through those channels, not through a single headline number. New firms built around such tools also face the ordinary business questions we cover in how startups actually make money (https://therealigned.com/story/how-startups-actually-make-money).

What is the strongest case against our reading?

We think that objection is fair, and our reading does not rebut it. Its limit is that it predicts, while the evidence we opened only describes the present state of the tests.

What is still not known?

Several things are open. Nobody has shown how much of the uncovered work current models could do, because the tests that would show it do not exist yet. The hours weights are model-generated estimates, not time-use measurements that we saw. Mercor's own check comparing simple and task-weighted scores for four occupations found them generally similar, but fewer than one-third of the relevant tasks had enough coverage to be included. And Mercor raises the question of whether tests that mainly measure automation generalise to the human-assisted use that is more common today, and says finding a method that scales remains elusive. ¹