AI system reconstructs viewed images from brain scans
Researchers at the Weizmann Institute of Science say their system can recreate images people see using high-resolution fMRI data, with about an hour of calibration for a new participant.
An AI system developed by researchers at Israel’s Weizmann Institute of Science reconstructs images people have viewed from high-resolution brain scans. It can also predict brain activity from an image, the team reports.
The system combines a brain decoder, which estimates an image’s structure and content from fMRI data, with an encoder that predicts brain activity from images. A diffusion model then helps produce the reconstruction. The researchers say about 70 per cent of the images used for training were not paired with fMRI scans.
The team reports that the system can be calibrated for a new person using about one hour of fMRI data, compared with roughly 40 hours for some earlier approaches. The work was presented at the Cognitive Computational Neuroscience conference in New York last month.
The reconstructions can be wrong: examples described by the researchers include a cake rendered as sandwiches and a dog in a bathtub rendered as a goat. The team says its system performed better than previously described tools, though the available account does not provide detailed results to assess that comparison independently.
Researchers hope to extend the approach to imagined scenes, video, audio and dreams, and to explore whether brain activity could help people who are locked in communicate. Those capabilities have not been demonstrated in the reported work, which focuses on images people have seen.
The research has also prompted questions about mental privacy. The current system uses fMRI, but scientists cited in the report warn that future methods using EEG could make it easier to infer mental imagery without consent.
