risk · R14
AI-Enabled Generation of Illegal and Harmful Content
The rapidly increasing use of generative AI to create and disseminate illegal or inherently harmful material, such as Child Sexual Abuse Material (CSAM) and non-consensual deepfake pornography. This is a widespread and escalating risk, moving beyond isolated incidents to a surge in content that is criminal and causes severe personal harm.
Updated
Observatory assessment: 74 / 100. Critical
Deep dive
The use of generative AI to create and disseminate illegal or inherently harmful material, such as Child Sexual Abuse Material (CSAM) and non-consensual deepfake pornography. This risk goes beyond disinformation to the direct creation of content that is criminal or causes severe personal harm.
Why this is a distinct topic: This is distinct from R03 (Disinformation), which focuses on false information, and k1klQA1yavTlSiscjMCI (Radicalization), which focuses on influencing behavior. This risk is about the creation of illicit artifacts themselves.
Key themes: generative AI misuse, non-consensual content, CSAM, synthetic media harm
Trajectory and projections
Mitigation strategies
- Platform-level content moderation and restrictions on AI generation capabilities.
- Development of in-generation safety detection and filtering technologies.
- Legal frameworks and law enforcement action against the creation and distribution of AI-generated illegal content.
- Public awareness campaigns and educational programs for potential victims, parents, and schools.
Top evidence
Yapay zeka videoları ile 33 kişiyi dolandırdılar. 15 milyon TL'lik vurgun
AI Incident Database
Suspects in Yozgat, Turkey, were apprehended for defrauding 33 individuals of approximately 15 million Turkish Liras using AI-generated videos, demonstrating the use of AI for sophisticated financial crime.
UPDATED: Officials: 3 Cascade students charged in connection with deepfake photos
AI Incident Database
Three high school students have been charged for using AI to generate non-consensual nude deepfake images of other students. This incident highlights the direct creation and dissemination of illegal and harmful content using generative AI, impacting children and their reputations.
Her childhood photo. Thousands of explicit images. One woman’s nightmare.
AI Incident Database
A woman discovered thousands of explicit images created from her childhood photos, leading to a legal investigation. This incident highlights the severe personal harm caused by the AI-enabled generation and dissemination of illegal and harmful content.
Deepfake pornovideo’s schokken medialandschap: BN’ers en politici doen aangifte
AI Incident Database
Dutch celebrities and politicians are filing complaints and seeking legal action after becoming victims of AI-generated deepfake pornography, prompting calls for a parliamentary debate and urging victims to report incidents to authorities.
Kamerleden horen al maanden niets na deepfake pornovideo's: 'Week ophef, daarna werd het stil'
AI Incident Database
Dutch Members of Parliament who were victims of deepfake pornography videos report a lack of follow-up on their complaints months after the initial incidents, highlighting potential gaps in the legal and enforcement response to AI-generated harmful content.