Coding agents struggle to use testing and verification tools effectively
Dan Luu’s evaluation found that coding agents often used testing libraries and formal methods superficially, while additional instructions increased work without reliably improving correctness.
Coding agents frequently fail to use testing and verification techniques effectively, according to an evaluation by Dan Luu published on 8 September and surfaced by Hacker News. The study examined agents implementing Rust versions of the Zstd and IMAP specifications, with performance assessed against hidden tests.
Across 26 testing and verification conditions, no technique consistently outperformed the others. Default prompting, with no additional testing instructions, performed above average, while a small custom testing skill recorded the highest score. Luu cautioned that the skill was not fully effective and that the differences between conditions may reflect substantial run-to-run variation.
Agents often used named libraries and formal methods without applying their core techniques. Formal tools were used to prove relatively unimportant or abstract properties, while property-based testing and fuzzing commonly relied on random or invalid inputs that covered few meaningful behaviours. Agents also produced tests that encoded incorrect results or failed to exercise risky cases.
Test-driven development increased the number of tests and led to a more iterative workflow, but did not materially improve correctness in the reported experiments. Several recommended skills similarly increased testing activity and cost without producing a clear benefit. One property-testing skill, for example, added substantial work while mostly generating additional checks that did not target the most error-prone areas.
There were occasional productive uses of fuzzing, property-based testing and shrinking, suggesting agents can apply these methods under some conditions. But the evaluation found that brief instructions to use a technique or library were generally insufficient. Luu said targeted human guidance, improved training or task-specific prompting could produce better results.
The findings are limited to the experiments, which used RFC-based Zstd and IMAP tasks that may not represent ordinary software development. The Zstd results averaged 80 runs per condition, while the IMAP results used smaller samples, and no independent replication was identified.
