Encord trials brainwave sensors to solve physical AI data bottleneck

Encord, a data tooling company based in San Leandro, California, is trialling the use of brainwave sensors to generate training data for physical AI models. In collaboration with German neuroscience startup Zander Labs, Encord’s robotic trainers wear headsets that measure brain activity to deduce mental states such as error, intent, and surprise. This initiative addresses the scarcity of real-world physical training data, which experts estimate needs to be five times the size of YouTube’s video corpus to advance humanoid and warehouse robotics. The trial aims to determine if brainwave-tagged data improves model performance before scaling.

Andrew Ceja, an Encord pilot, is observed wearing a Zander Labs headset while performing tasks such as disassembling a Jenga tower in the company’s San Leandro warehouse. The headset includes sensors that measure brain waves to provide clues for model builders trying to figure out when they need to deploy their highest-effort models. Lucas Gehrke, a Zander neuroscientist supervising the work, notes that the amount of brain activity used at any point during a task offers valuable insights for training.

Sofia Infante, another pilot, is using leader-follower rigs to manipulate robotic arms for tasks including plugging and unplugging ethernet cables. These rigs consist of paired robotic arms where one is controlled by a human and the other mimics the movements, creating data for tasks like pouring coffee and stacking poker chips. Infante and Ceja are part of a workforce that previously worked at Scale, another AI data annotation firm, before joining Encord.

Encord is developing a new data modality using forearm sensors to detect electrical signals in muscles, aiming to create a 3D depiction of hand positions that video alone cannot capture. The company is producing dense annotations for its data sets, describing physical actions such as “right hand tightens bolt,” which the company estimates is worth 100 times more than standard ego-centric video for specific tasks.

Vineeth Velmurugan, Encord’s head of robot learning, is a veteran of OpenAI’s robot lab and Berkshire Grey. He joined Encord to build the company’s internal data-creation team after executives at customer firms realised they had to produce training data themselves rather than simply manage it. Velmurugan describes the brainwave trial as the “bleeding edge” of the effort to solve the robotics data bottleneck.

The broader industry context involves a shift from managing existing data to manufacturing new data, as generative AI models for robots require significantly more physical training data than text-based LLMs. Unlike text, which can be scraped from the internet at minimal cost, physical training data must be manufactured, changing the economics of building these models. Encord’s customers include several leading robotics firms, though specific names are not authorised for disclosure.

Encord’s work with Zander is currently a trial run; the company says the goal is to build an initial brainwave-tagged data set, run it through customer robotics models, and evaluate whether it actually improves performance before deciding whether to scale it up. Encord does both egocentric video collection and remote robot operation, drawing data from factories around the globe while experimenting with new modalities at its San Leandro facility.