$OneMillion-Bench: How Far are Language Agents from Human Experts?

Qianyu Yang^*, Yang Liu^*, Jiaqi Li^*, Jun Bai^*, Hao Chen, Kaiyuan Chen, Tiliang Duan, Jiayun Dong, Xiaobo Hu, Zixia Jia, Yang Liu, Tao Peng, Yixin Ren, Ran Tian, Zaiyuan Wang, Yanglihong Xiao, Gang Yao, Lingyue Yin, Ge Zhang, Chun Zhang, Jianpeng Jiao^#, Zilong Zheng^#, and Yuan Gong^#

2026 · arXiv: arxiv.org/abs/2603.07980

Code Website

Abstract

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focused on expert-level problems to ensure meaningful differentiation across agents. Together, $OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios.

Citation

@misc{yang2026onemillionbench,
      title={\$OneMillion-Bench: How Far are Language Agents from Human Experts?}, 
      author={Qianyu Yang and Yang Liu and Jiaqi Li and Jun Bai and Hao Chen and Kaiyuan Chen and Tiliang Duan and Jiayun Dong and Xiaobo Hu and Zixia Jia and Yang Liu and Tao Peng and Yixin Ren and Ran Tian and Zaiyuan Wang and Yanglihong Xiao and Gang Yao and Lingyue Yin and Ge Zhang and Chun Zhang and Jianpeng Jiao and Zilong Zheng and Yuan Gong},
      year={2026},
      eprint={2603.07980},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.07980}, 
}