AI Benchmarks Miss 90% of US Work Hours, New Study Finds

AI Benchmarks Overlook Most Real-World Work, New Analysis Shows

Artificial intelligence benchmarks have proliferated, yet a new analysis from Mercor indicates they barely scratch the surface of the US economy. The research, which maps 29 leading AI benchmarks against the Department of Labor's O*NET occupational database, found that only 208 of 923 occupations have tasks covered by at least five benchmark prompts. More strikingly, an estimated 90% of American working hours are spent on tasks with little or no benchmark representation.

This coverage gap is not random. The study reveals a systematic bias: benchmarks disproportionately target tasks in high-wage, AI-exposed occupations. While this focus might seem logical initially, it creates significant blind spots for mid-tier knowledge work, physical roles, and the long tail of the economy. The result is a distorted picture of AI's true economic capabilities and limitations.

Mapping the Disconnect Between Benchmarks and Jobs

Mercor's Benchmark Map is built on the O*NET taxonomy, which breaks the US workforce into 923 detailed occupations encompassing 18,797 distinct tasks. The team selected 29 benchmarks based on public availability, agentic task focus, economic realism, and balanced difficulty. An LLM judge then tagged each benchmark prompt with corresponding O*NET tasks, validated by human experts for Mercor's own APEX benchmarks.

The findings are stark. Only 22% of occupations have any task covered by at least five benchmark prompts, and six of the 22 major job families have zero meaningful coverage. The analysis shows that tasks rated as 'AI-exposed' are roughly six times more likely to be covered than others. Even after controlling for AI exposure, occupations with larger wage bills receive significantly more benchmark attention.

Why High-Wage, AI-Exposed Jobs Dominate Benchmarks

The pattern is clear: benchmarks are concentrated in areas like software engineering and financial analysis, while truck driving and gas plant operation are virtually absent. This is partly a rational allocation of limited resources, but it has a downside. For example, lawyers' tasks are heavily benchmarked, but paralegals and court clerks—who perform similar AI-exposed work—are largely ignored. This creates a skewed view of where AI can truly add value.

This bias extends to task importance. Even within the same occupation, tasks rated as most important are 6-10 times more likely to be covered than those rated least important. While prioritizing key tasks makes sense, it means that less critical but time-consuming duties are overlooked, which can lead to overestimating AI's usefulness in a given role.

The Challenge of Measuring AI Performance by Occupation

Mercor also explored whether it's possible to generate accurate occupation-level AI scores by weighting benchmark results with O*NET task importance. Their initial tests on four occupations—investment banking, law, accounting, and software engineering—showed that simple averages and O*NET-weighted scores were broadly similar. However, less than a third of relevant O*NET tasks had enough benchmark prompts to include in the weighted scores.

This highlights several persistent obstacles. First, the statistical power is inherently low given the breadth of US jobs. Second, benchmark difficulty and quality vary wildly, with some tasks saturated and others unsolved. Third, most benchmarks measure full automation, not the augmentation that characterizes most real-world AI use. Finally, roughly 63% of jobs involve physical components that current text-based benchmarks cannot address.

Real-World Performance Gaps Beyond Benchmarks

The benchmark coverage problem is compounded by performance gaps within covered tasks. A separate benchmark, LSCBench, tested six frontier coding agents on 450 tasks derived from real-world vulnerabilities. The best agent passed 91.3% of functional tests but only 25.1% when security was also assessed. This illustrates that even in well-benchmarked areas, current metrics often miss critical dimensions of job performance.

Pew Research's AI exposure analysis further underscores the nuance. Paralegals rank high on exposure due to document-heavy work, but courtroom advocacy and client counseling remain firmly human tasks. Similarly, construction and transportation jobs rank low because they require physical presence in unpredictable environments. These distinctions are lost when benchmarks focus narrowly on text-based, high-wage tasks.

The Expectations Gap Between AI Developers and Workers

Anthropic's research highlights a widening chasm between what AI developers and the public expect. While 91% of Anthropic staff expect AI to achieve Nobel-level discoveries by 2030, only 39% of US adults agree. More tellingly, over half of staff expect AI to handle 80% of their job by 2030, versus just 5% of US workers. This disconnect may stem from developers benchmarking tasks they know well, while workers see the messy, physical, and interpersonal parts of their jobs that benchmarks ignore.

This expectations gap has practical consequences. Companies risk investing in AI for tasks that benchmarks suggest are easy but that fail in real-world contexts, or, conversely, they may underutilize AI where it could genuinely help. The Mercor analysis suggests that until benchmarks diversify to cover the full spectrum of economic activity, our understanding of AI's true impact will remain dangerously incomplete.

Toward a More Representative Benchmark Ecosystem

Mercor's work is a first step, and the researchers acknowledge limitations. Their list of 29 benchmarks is not exhaustive, and selection criteria like 'economic realism' are subjective. However, the core finding—that 90% of working hours are spent on under-benchmarked tasks—is a powerful call to action for the AI community to develop more economically diverse and realistic benchmarks.

As AI continues to permeate the workforce, the need for accurate capability mapping becomes critical. Policymakers, enterprises, and workers all rely on these measurements to make informed decisions. Bridging the benchmark gap is not just an academic exercise; it is essential for ensuring AI is deployed where it genuinely helps, and for setting realistic expectations about what automation can and cannot achieve.