Google debuts Gemini AI avatar tool, raising deepfake concerns despite safety guardrails
While Google restricts usage to self-portraiture and cites safety priorities, the launch of the Omni video model model highlights ongoing tensions between generative AI innovation and nonconsensual content risks.
Google has launched a new avatar feature within its Gemini app, powered by the Omni video model, enabling subscribers to generate short videos featuring a digital clone of themselves. The tool requires users to record a short video of their face to create the likeness. While Google states the feature is restricted to users creating content of their own likeness, the technology raises concerns regarding nonconsensual deepfakes. Early testing revealed the generated videos were photorealistic but contained logical errors and uncanny visual details.
The avatar feature is exclusively available to Google AI Pro subscribers, costing $20 per month, with usage limits that reset every five hours. To create an avatar, users must record a short video of their face in a well-lit room, reading two-digit numbers and swivelling their head, a process taking approximately five minutes. The avatar feature currently restricts users to generating content only of their own likeness, unlike previous iterations of similar technology that allowed for broader likeness usage.
Early testing by a WIRED reporter demonstrated the tool’s capability to generate photorealistic but logically inconsistent clips, such as the reporter singing to a dinosaur in Dolores Park and surfing under the Golden Gate Bridge while wearing denim. Google’s strength in mapping and location data contributes to the photorealistic accuracy of backgrounds in generated videos, such as Dolores Park in San Francisco.
The technology is comparable to features previously offered by OpenAI’s now-defunct Sora app. Generative AI tools without strict guardrails have increasingly been used to create nonconsensual deepfakes, particularly targeting women. Google DeepMind’s Nicole Brichtova stated that the company prioritises safety and aims to prevent harm while not blocking benign uses.
The long-term scalability and stability of the Omni video model under high demand are not yet fully established, as evidenced by immediate usage limits. The effectiveness of Google’s safety measures in preventing misuse of the avatar tool for nonconsensual purposes remains to be seen, despite current restrictions. Claims regarding the "photorealistic" nature of the output should be contextualised with reports of visual errors, such as misaligned teeth and nonsensical outfits.
The assertion that the tool prevents harm is a corporate claim; independent verification of safety protocols is not provided in the source material. Descriptions of the generated content as "creepy" or "unnerving" reflect subjective user experience rather than objective technical metrics.
