AI Impersonation and Flawed Automation Cause Real-World Harm
Published
Executive summary
This week's intelligence brief reveals a critical escalation in AI's capacity for identity appropriation and misinformation, with deepfakes impersonating doctors spreading health misinformation and a political party embroiled in a deepfake scandal. Concurrently, incidents of AI generating fake legal citations and misguiding public policy underscore a concerning rise in uncontrolled agentic actions and a loss of human oversight. These developments highlight accelerating risks related to identity integrity and the reliability of AI in critical sectors.
Outlook
Expect continued challenges in verifying digital authenticity and managing autonomous AI systems, as these pervasive risks are set to accelerate into the coming weeks.
Top signals
AI Deepfakes of Real Doctors Spreading Health Misinformation on Social Media
An investigation revealed hundreds of AI-generated deepfake videos impersonating real doctors to spread health misinformation and sell unproven supplements on social media. This poses a significant public health threat by eroding trust in medical professionals and amplifying disinformation.
Source: AI Incident Database
Class Action Alleges That Grammarly Misappropriated the Names of Journalists and Authors Through its “Expert Review” That Lets Users Get Feedback on Writing From Experts
A class action lawsuit alleges Grammarly misappropriated the names and identities of journalists and authors for its "Expert Review" feature without consent. This case underscores the legal and ethical challenges posed by AI systems that replicate human personas for commercial purposes, infringing on privacy and intellectual property rights.
Source: AI Incident Database
„Gerne hätten wir noch eindeutiger Stellung bezogen“ – So erklärt die CDU intern die Deepfake-Affäre
The CDU in Lower Saxony, Germany, is embroiled in a deepfake scandal, prompting the state leader to address members internally. This incident highlights the direct impact of AI-amplified disinformation on democratic institutions and political processes, eroding trust and distorting public discourse.
Source: AI Incident Database
Sixth Circuit Sanctions Attorneys for Fake Citations – What Does This Mean for Use of AI?
The Sixth Circuit sanctioned two Tennessee attorneys for using artificial intelligence to generate fake legal citations, leading to misconduct during merits briefing. This case underscores the dangers of AI hallucinations and the importance of human oversight in legal practice.
Source: AI Incident Database
Are China’s ‘AI tigers’ cheating? US rival Anthropic alleges some are
U.S. AI firm Anthropic accuses three prominent Chinese AI labs of illegally extracting capabilities from its Claude model, citing national security concerns. This highlights the growing risks of intellectual property infringement and potential misuse of advanced AI models in an international context.
Source: AI Incident Database
How DOGE Gutted the NEH in 22 Days
The Department of Government Efficiency reportedly used ChatGPT to identify National Endowment of the Humanities grants for cancellation, leading to significant cuts. This incident highlights the risks of delegating critical policy decisions to AI without adequate human oversight, potentially leading to biased or misinformed outcomes.
Source: AI Incident Database
Topics in the news
Autonomous Weapons and AI-Enabled Conflict Escalation
41 signals / 30 days
AI Power Concentration and Oligopoly Control
31 signals / 30 days
AI System Integrity Failure from Misaligned Incentives
31 signals / 30 days
AI-Enabled Mass Surveillance and Authoritarian Governance
29 signals / 30 days
Loss of Human Agency and Cognitive Atrophy
28 signals / 30 days
Assessment updates
Observatory assessments, not measured outcomes.
Emergent AI Social Systems and Governance Failure
2026 score: 45 → 50
Governance gaps widening
Emergent AI Social Systems and Governance Failure
Risk velocity: Accelerating → High
Governance gaps widening
Unauthorized AI Appropriation of Identity and Persona
2026 score: 75 → 82
Extraordinary signal density
Unauthorized AI Appropriation of Identity and Persona
2035 projection: 85 → 90
Extraordinary signal density
Unauthorized AI Appropriation of Identity and Persona
Risk velocity: Accelerating → Critical
Extraordinary signal density
Digital Content Provenance (C2PA) Standards
2026 score: 40 → 48
Inflection point
Digital Content Provenance (C2PA) Standards
2035 projection: 85 → 87
Inflection point
Digital Content Provenance (C2PA) Standards
Adoption stage: Early Adoption → Scaling
Inflection point
Emergent Unpredictable Behavior in Autonomous Agent Systems
2026 score: 45 → 55
Safety non-compositionality proven
Emergent Unpredictable Behavior in Autonomous Agent Systems
Risk velocity: Accelerating → High
Safety non-compositionality proven
Model Collapse and Data Scarcity Crisis
2026 score: 60 → 63
Data contamination at crisis level
AI for Public Health, Safety, and Accessibility
2026 score: 20 → 35
Billion-dollar deals and FDA engagement
AI for Public Health, Safety, and Accessibility
2035 projection: 50 → 60
Billion-dollar deals and FDA engagement
AI for Public Health, Safety, and Accessibility
Adoption stage: Research → Pilot
Billion-dollar deals and FDA engagement
AI-Induced Psychological and Relational Harm
2026 score: 65 → 72
Documented financial devastation
AI-Induced Psychological and Relational Harm
2035 projection: 85 → 88
Documented financial devastation
AI-Induced Psychological and Relational Harm
Risk velocity: Accelerating → Critical
Documented financial devastation
Societal Backlash and Resistance to AI
2026 score: 65 → 70
Backlash driving corporate actions
Societal Backlash and Resistance to AI
Risk velocity: Accelerating → High
Backlash driving corporate actions
Environmental Cost of AI Infrastructure
2026 score: 68 → 72
UK CO2 hundreds of times higher than thought
Environmental Cost of AI Infrastructure
2035 projection: 75 → 78
UK CO2 hundreds of times higher than thought
Autonomous Weapons and AI-Enabled Conflict Escalation
2026 score: 82 → 85
AI actively used in military operations
Negative Socio-Economic Impacts of AI Infrastructure
2026 score: 65 → 70
Environmental impact exceeding estimates
Negative Socio-Economic Impacts of AI Infrastructure
Risk velocity: Accelerating → High
Environmental impact exceeding estimates
Privacy-Preserving AI: Federated Learning & On-Device Processing
2026 score: 30 → 35
Landmark privacy-preserving AI at scale
Privacy-Preserving AI: Federated Learning & On-Device Processing
2035 projection: 75 → 77
Landmark privacy-preserving AI at scale
Algorithmic Auditing & Fairness Certification Standards
2026 score: 30 → 33
New frameworks
Algorithmic Auditing & Fairness Certification Standards
2035 projection: 70 → 72
New frameworks
International AI Arms Control Treaties
2026 score: 10 → 15
Pentagon saga catalyzed legislation
International AI Arms Control Treaties
Adoption stage: Negotiation → Policy Debate
Pentagon saga catalyzed legislation
AI Model Integrity and Supply Chain Vulnerabilities
2026 score: 65 → 70
Active supply chain attacks
AI Model Integrity and Supply Chain Vulnerabilities
Risk velocity: Accelerating → High
Active supply chain attacks
AI-Facilitated Violence and Radicalization
Risk velocity: Accelerating → Medium
Zero signals, velocity downgrade
AI Power Concentration and Oligopoly Control
2026 score: 82 → 85
Massive capital consolidation
Open-Source AI & Antitrust Enforcement
2026 score: 40 → 45
Open-weight ecosystem growing
Systemic Algorithmic Discrimination in Critical Sectors
2026 score: 83 → 86
Multiple Critical-severity real-world incidents
Systemic Algorithmic Discrimination in Critical Sectors
2035 projection: 68 → 71
Multiple Critical-severity real-world incidents
Emergent AI Societal Structures and Unpredictable Behavior
2026 score: 35 → 40
Autonomous deployments and formal proofs
Emergent AI Societal Structures and Unpredictable Behavior
Risk velocity: Accelerating → High
Autonomous deployments and formal proofs
Corporate Ethical Red Lines & Developer-Led Governance
2026 score: 40 → 48
Legal precedents and commercial safety tools
Democratic AI Oversight & Surveillance Moratoriums
2026 score: 20 → 28
Multi-branch democratic engagement
Democratic AI Oversight & Surveillance Moratoriums
2035 projection: 55 → 60
Multi-branch democratic engagement
Democratic AI Oversight & Surveillance Moratoriums
Adoption stage: Advocacy → Policy Debate
Multi-branch democratic engagement
AI for Environmental and Urban Resilience
2026 score: 0 → 25
Severely underscored
AI for Environmental and Urban Resilience
2035 projection: 0 → 50
Severely underscored
AI for Environmental and Urban Resilience
Adoption stage: Research → Pilot
Severely underscored
AI Deception and Strategic Obfuscation
2026 score: 45 → 55
First production model scheming
AI Deception and Strategic Obfuscation
2035 projection: 80 → 85
First production model scheming
AI Deception and Strategic Obfuscation
Risk velocity: Accelerating → Critical
First production model scheming
Uncontrolled Real-World Actions by Agentic AI
2026 score: 50 → 60
Real-world agentic failures
Uncontrolled Real-World Actions by Agentic AI
2035 projection: 50 → 58
Real-world agentic failures
Uncontrolled Real-World Actions by Agentic AI
Risk velocity: Medium → Critical
Real-world agentic failures
Adversarial Testing and Red Teaming for AI Safety
2026 score: 20 → 32
Transitioning to commercial tooling
Adversarial Testing and Red Teaming for AI Safety
2035 projection: 50 → 58
Transitioning to commercial tooling
Adversarial Testing and Red Teaming for AI Safety
Adoption stage: Research → Pilot
Transitioning to commercial tooling
Loss of Human Agency and Cognitive Atrophy
2026 score: 65 → 70
Risk transitioned to documented harms
AI-Enabled Generation of Illegal and Harmful Content
2026 score: 50 → 62
Active crisis with legal consequences
AI-Enabled Generation of Illegal and Harmful Content
2035 projection: 50 → 65
Active crisis with legal consequences
AI-Enabled Generation of Illegal and Harmful Content
Risk velocity: Medium → Critical
Active crisis with legal consequences
Universal Basic Services & AI-Era Workforce Transition Programs
2026 score: 15 → 18
First US legislative mechanism
Universal Basic Services & AI-Era Workforce Transition Programs
2035 projection: 45 → 47
First US legislative mechanism
Autonomous AI-Driven Cyberattacks and Malware
2026 score: 65 → 72
AI-powered attacks at scale
Autonomous AI-Driven Cyberattacks and Malware
Risk velocity: Accelerating → Critical
AI-powered attacks at scale
Green AI Standards & Carbon-Aware Computing
2026 score: 30 → 35
Legislative action + efficiency breakthroughs
AI System Integrity Failure from Misaligned Incentives
2026 score: 65 → 72
Behavioral degradation outpacing mitigation
AI System Integrity Failure from Misaligned Incentives
Risk velocity: Accelerating → High
Behavioral degradation outpacing mitigation
Mass Labor Displacement and Economic Polarization
2026 score: 85 → 89
Landmark mass layoffs attributed to AI
Mass Labor Displacement and Economic Polarization
2035 projection: 70 → 75
Landmark mass layoffs attributed to AI
Mass Labor Displacement and Economic Polarization
Risk velocity: High → Critical
Landmark mass layoffs attributed to AI