San Francisco demands Apple and Google remove nudify apps amid deepfake concerns
The city’s legal office has issued cease-and-desist letters targeting 13 applications capable of generating non-consensual intimate images, arguing their presence violates California’s prohibitions on deepfake pornography.
San Francisco Attorney General David Chiu has issued cease-and-desist letters to Apple and Google, demanding the immediate removal of 13 applications from their respective app stores that are capable of generating non-consensual intimate images. The legal action targets so-called nudification apps, which allow users to trivially transform ordinary photographs of real people into explicit images by removing clothing, altering features, or swapping faces onto naked bodies. Chiu’s office argues that the continued availability of these tools violates California laws prohibiting services that support the creation of deepfake pornography.
The Attorney General’s office estimates that Apple and Google have likely profited millions of dollars from these applications through fees, arguing that the tech giants have failed to take stronger actions or develop adequate detection mechanisms to avoid profiting from what Chiu described as a public nuisance. Chiu told Wired that his office was “absolutely horrified” by the ubiquity of the technology, noting that it has victimised mostly women and children at an alarming scale, leading to bullying, humiliation, and severe mental health impacts, including suicidal ideation among victims.
Wired reviewed the letters and confirmed that Chiu asked Google to remove five specific apps and Apple to remove eight. No specific app names were included in the correspondence to avoid directing users toward them, although one application reportedly had more than a million downloads and advertised features to sexualise images of women. Chiu stated that allowing any such apps to remain in app stores is unacceptable, emphasising that generating non-consensual intimate images is illegal, harmful, and completely unacceptable.
In response to the demands, Google confirmed that five of the flagged apps have been suspended from the Google Play store for violating policies against harmful content. A spokesperson for Google stated that the platform does not allow apps containing sexual content and that they take proactive steps to detect and remove such material, including suspending hundreds of violating apps and restricting related search terms. Apple, however, has not yet responded to the request or to Ars Technica’s request for comment, maintaining a notably silent stance despite previous private threats to remove other controversial applications.
The regulatory pressure comes amidst broader concerns about generative AI tools, including xAI’s Grok, being used to create non-consensual intimate imagery and child sexual abuse material. xAI recently filed a lawsuit confirming it found instances of Grok-generated illegal content, raising questions about liability and app store policy enforcement. Although Chiu’s letters did not explicitly request the removal of Grok, he indicated that his demands are intended to help tighten up enforcement across both platforms. He warned that if the companies do not remove the flagged apps and strengthen their screening systems, the city will have to consider all its legal options.
