risk · R03

AI-Amplified Disinformation and Reality Distortion

AI tools have lowered the cost of generating convincing disinformation to near zero, now extending beyond deepfakes to include poisoning AI search results and enabling sophisticated cybercrime. The emergence of autonomous agents generating harmful content and AI-coordinated human propaganda ('cyborg propaganda') represents a new, more complex phase of this risk.

Updated

Observatory assessment: 98 / 100. Critical

How assessments are made

Deep dive

The 'Liars Dividend' has arrived, but the threat has evolved beyond simple content generation. New vectors include 'cyborg propaganda,' where AI optimizes and coordinates human actors to spread partisan messaging [X4N10ZKtW4igQgIhrF3W]. Adversaries are directly attacking AI information systems by poisoning the data used by search tools (RAG systems) to inject scams and misinformation [BY9xxwF1yXqhnP8GnubX]. This risk has also rapidly commercialized into widespread criminal fraud, with deepfakes of public figures used globally to promote health and financial scams [WVVLxsSr4MOvm4CKK3vU, 0QmQZr0ragCPB0ls3UUv]. Concurrently, fundamental model vulnerabilities are becoming more sophisticated, with research demonstrating models can harbor 'narrow secret loyalties' that are difficult to detect with standard audits [HA7VrqpuhTVfn5W21roO], making the problem more systemic and harder to contain. The risk continues to manifest in novel ways, including AI-voice-cloned 'swatting' calls causing real-world police responses [YkXMrU8VjmyU7We0xTxj], the use of AI in defamatory political ads [X4nJrvOUBNluKQIlhMjs], and the overwhelming of academic peer-review systems with AI-assisted papers, threatening scientific integrity [YvSIpQursAbJLJQAtUNt].

Trajectory and projections

Near term

By 2026, 'Reality Apathy' sets in; the public stops trusting all digital media. Biometric verification for social media accounts becomes a major political debate.

Mid term

By 2030, the internet fractures into 'Authenticated' zones (verified humans) and 'The Wild' (AI bots). News is cryptographically signed at the source (C2PA).

Long term

By 2035, AI 'Truth Defense' agents automatically flag manipulation in real-time AR glasses. Disinformation shifts to closed, encrypted networks.

Mitigation strategies

  • Implementation of C2PA / Watermarking standards at the camera hardware level
  • AI-literacy education as a mandatory school curriculum
  • Liability laws for platforms amplifying non-labeled synthetic content
  • Cryptographic 'Personhood Credentials' for online interaction
  • Technical defenses for AI systems (e.g., robust RAG architectures)
  • Development of verification standards and professional ethics guidelines for AI use in specialized domains (e.g., law, academia, journalism)
  • Comprehensive platform regulation (e.g., EU's Digital Services Act) targeting systemic risks and algorithmic design
  • Development of standardized benchmarks and red-teaming frameworks for information operations (e.g., InfoOps Bench)

Top evidence

Commission starts enforcing AI Act rules and new transparency requirements on 2 August

EU Digital Strategy (EC)

The European Commission's AI Office will begin enforcing the AI Act on August 2nd, alongside new transparency rules requiring AI systems to disclose when users are interacting with AI or when content has been generated or altered by it.

S. Hrg. 119-513 - THE AI DECEPTION MACHINE: DEEPFAKES, CHATBOTS, AND THE NEW FRONTIER OF SENIOR FRAUD

GovInfo Congressional Hearings

This congressional hearing title indicates a focus on how AI, specifically deepfakes and chatbots, is being used to perpetrate new forms of fraud targeting seniors. It highlights the emerging threat of AI-enabled deception in financial crime.

The Influence Operation That Ran on Borrowed Reputations

Council on Foreign Relations

A Russia-linked influence operation utilized ChatGPT to construct a credible-looking fake research institute, complete with fabricated papers and real experts listed without consent, demonstrating the advanced capabilities of AI in generating convincing disinformation and appropriating identities.

‘Too good to be true?’: No, WHYY’s Terry Gross did not personally email you to be a guest on ‘Fresh Air’

AI Incident Database

An elaborate scam is using the names of celebrity podcast hosts like Terry Gross and Nikki Glaser to trick authors and performers into believing they are invited for interviews. This incident highlights how AI tools can facilitate sophisticated identity appropriation and disinformation campaigns, making it easier to create convincing but fraudulent communications.

AI hallucinated judgments: Why Supreme Court set aside a tribunal order

AI Incident Database

An Indian Supreme Court set aside a tribunal order because it was based on six AI-hallucinated judgments, three of which were entirely fabricated and three were misrepresented. This incident highlights the critical risks of AI generating false information in legal contexts.