Nvidia research suggests AI harness outperforms model choice in long-horizon tasks
New findings from Nvidia indicate that the scaffolding surrounding an AI model is more critical to performance than the model itself, with a custom setup achieving a perfect score on a key reasoning benchmark.
Nvidia has published new research indicating that the "harness"—the scaffolding, tools, and runtime environment surrounding an AI model—is often more critical to performance than the model itself, particularly for long-horizon tasks. The study demonstrates that the infrastructure managing memory, context, and feedback can determine whether an AI agent maintains focus over extended periods or deviates from its objective.
In the study, Nvidia researchers achieved a 100 per cent score on the ARC-AGI-3 interactive reasoning benchmark using the Claude Opus 5 model. This perfect score was achieved by implementing a custom harness known as Agentic Variation Operators (AVO), which includes a "supervisor" component. Without this specific harness configuration, the same model scored only 30 per cent, which was the highest result among all models tested in the study.
The ARC-AGI-3 benchmark consists of a suite of 2D games with no instructions, requiring the model to figure out how to play and win. A 100 per cent score indicates performance on par with humans. Nvidia Vice President of Product Adel El Hallack explained that the harness handles memory, context, and feedback, distinguishing it from the model, which acts as the "brain". He noted that while the world often interprets an agent as an API of the model, it is actually the combination of the model and the scaffolding around it.
The custom AVO harness includes a supervising agent that acts like a CEO, nudging the main agent when it deviates or hits a dead end. El Hallack stated that this supervisory component prods the agent in the right direction if it starts exploring a path that might lead to a dead end or re-explores a previously trodden path. While the concept of a supervising agent is not new, most current agent users rely on only one layer for their harness, such as Claude Code or Codex.
This finding aligns with previous industry observations regarding the limitations of large language models in complex tasks. In April, Microsoft research found that 19 large language models, including frontier ones, filled documents with errors when performing long-horizon editing tasks. Similarly, OpenAI previously conducted its own research on ARC-AGI-3 after its models scored less than 10 per cent, finding that tweaking harness settings tripled their scores, though none reached 100 per cent.
Nvidia’s results add to the growing evidence that model choice is far from the only factor in agentic performance and cost. In July, Databricks CEO Ali Ghodsi noted that the choice of harness can double AI costs compared to the model choice alone. El Hallack added that open harnesses allow users to adjust more variables to improve accuracy, contrasting with closed systems. He argued that an open agent stack, where users have control across the harness, infrastructure, and runtime, is required to move the ecosystem forward securely.
