Waymo unveils 'Reference Driver' benchmark amid safety scrutiny

Alphabet-owned Waymo has published a research paper in Nature Communications detailing the 'Reference Driver', a new computer model designed to benchmark autonomous driving performance against human behaviour. Developed in conjunction with TU Delft, the model utilises an 'active inference' framework to simulate how human drivers anticipate and react to traffic conflicts, offering a more accurate assessment of robotaxi safety than previous reactive models. The company is releasing the research code under a non-commercial academic license to facilitate further study.

This development follows increased regulatory scrutiny after a January incident in Santa Monica, California, where a Waymo vehicle struck a child. During the incident, Waymo relied on its previous computer model to claim that an attentive human driver would have made impact at around 14 miles per hour, whereas the robotaxi hit the child at 6 miles per hour after decelerating from 17 miles per hour. The crash is currently under investigation by the National Highway Traffic Safety Administration and the National Transportation Safety Board.

The Reference Driver differs significantly from its predecessor by moving beyond last-second, reactive maneuvers. Instead, it simulates the internal 'surprise' a driver feels during a conflict, reproducing human behaviour in the run-up to a crash. Arkady Zgonnikov, an assistant professor at TU Delft, noted that this approach provides a more human-like benchmark for autonomous driving systems that was previously impossible to automate at scale.

Waymo stated that the new model is more accurate than the version used over the past several years. It can be adapted to represent a wide range of road user behaviours beyond collision avoidance and is better equipped for large test sets with thousands of scenarios. The company described the model as evolving the concept of crash dummies, serving as a behavioural benchmark for autonomous driving systems able to realistically represent reasonable expectations on how a careful human driver responds to traffic conflicts.

The automotive industry has historically used physical and virtual crash dummies to evaluate hardware and structural integrity. Waymo argues that the Reference Driver allows for the evaluation of numerous complex, real-world crashes in a virtual environment, identifying performance improvements with unprecedented speed and efficiency. The company is scaling operations to more cities and views this accurate modelling of human driving behaviour as essential for grading robotaxi performance.

Waymo is making the research code for the Reference Driver available under an academic, non-commercial license. This allows the code to be used for research, teaching, personal experimentation, and scientific publication, with the company hoping to collaborate with others to push the model further. The release aims to facilitate independent study and improve safety assessments as the technology faces greater public and regulatory attention.