Generalist AI Robots Learn New Tasks From Short Videos
Cambridge-based startup demonstrates robotic arms that improvise and adapt to new chores without specific task-based training, achieving a 59 per cent success rate.
A startup based in Cambridge, Massachusetts, has demonstrated robotic arms capable of learning new tasks from short instructional videos without requiring specific training for each action. Generalist AI, co-founded by Pete Florence, Andrew Barry, and Andy Zeng, showcased the technology during a visit by WIRED, where the robots performed simple chores such as stacking cups and manipulating objects. The demonstration highlighted the machines' ability to improvise when expected tools were missing or unavailable, a capability that distinguishes them from traditional AI-powered robots.
One notable example involved a robot instructed to sweep a block into a bowl using a dustpan and brush. When the brush was removed from the scene, the robot improvised by using the dustpan like a brush to flick the block into the bowl. In another instance, a two-armed robot watched a video of a person unzipping a purse to retrieve banknotes. When the robot attempted the task with a different type of purse and failed to grab the money with its right gripper, it switched to its left gripper to gain a better angle, successfully retrieving the notes.
The company’s approach focuses on teaching robots the physics of the world, drawing inspiration from how humans and children develop intuitive physical intelligence. This method contrasts with traditional robot training, which typically involves feeding thousands of examples into a model and often struggles with changes in lighting or environment. Generalist AI builds its AI models entirely from scratch rather than relying on open-source language models, aiming to create a general robotic model that can transfer learned skills across different scenarios.
To train these models, the company uses special gloves resembling robot pincers with attached cameras to collect physical interaction data from human workers. During the visit, a crate containing several hundred of these grippers was visible, destined for workers in Mexico and other locations. While the company has not disclosed the exact recipe for its training methodology, it claims to have gathered a significant amount of high-quality data. The co-founders, who previously worked at Google DeepMind and Boston Dynamics, have emphasised the importance of this data collection process in developing robust robotic capabilities.
Experts have noted the potential of Generalist AI’s approach for commercial deployment, particularly in manufacturing. Danfei Xu, a roboticist at Georgia Tech, stated that the startup has pushed the boundaries of general robot models and executed its strategy effectively. Karen Liu, a roboticist at Stanford University, added that the company’s method of collecting physical interaction data at scale without tying it to a specific robot suggests that their approach may be working. However, both experts acknowledged that the long-term reliability of these models in real-world settings remains to be proven.
Currently, the robots complete demonstrated tasks with an average success rate of 59 per cent, a figure the company acknowledges is not yet reliable enough for widespread commercial use. Ideally, the success rate would need to be upwards of 99 per cent for practical deployment. It is also unclear how well the robots’ skills will generalise to every imaginable task or setting. Despite these limitations, the ability of the robots to learn quickly and improvise has generated interest in the potential for future applications in various industries.
