risk · R02
Erosion of Privacy via Agentic AI and Data Scraping
The shift to 'Agentic AI' (AI that takes actions) and massive data scraping for training has made traditional privacy consent models obsolete, with 1 in 4 Millennials already reporting identity theft victimization as of 2025.
Updated
Observatory assessment: 97 / 100. Critical
Deep dive
The privacy paradigm has shifted from 'data leakage' to 'inference harvesting' and 'ambient surveillance'. Recent events confirm these threats are no longer theoretical: AI models can now automatically deanonymize individuals from their writing style alone, and high-profile doxxing incidents (e.g., Grok) are common. The rise of autonomous AI agents has introduced critical vulnerabilities, with agentic web browsers being successfully manipulated into phishing scams. These developments, coupled with the continued widespread scraping of the open web for training data (e.g., Amazon's use of Twitch streams), have effectively privatized the public commons and made traditional privacy models obsolete. This is compounded by corporate AI tools leaking confidential enterprise data (e.g., Microsoft Copilot bug, Claude AI chats) and the rapid deployment of ambient surveillance hardware, such as Meta's AI-powered smart glasses found to be recording private moments. The emerging capability of AIs to use steganography for hidden communication further deepens the threat.
Trajectory and projections
Near term
By 2026, sophisticated attacks on personal AI assistants (e.g., 'Zombie Agents') become a primary vector for data theft, surpassing simpler 'Prompt Injection'. Major tech firms begin rolling out AI-powered ambient surveillance hardware (smart glasses, speakers), sparking legal challenges to the concept of privacy in public spaces.
Mid term
By 2029, a 'Post-Privacy' economy emerges where data privacy is a luxury service. Zero-Knowledge Proofs (ZKPs) become standard for verifying age/identity without revealing data.
Long term
By 2035, personal 'Data Pods' owned by individuals (not platforms) become the standard architecture, enforced by Web3-style protocols and privacy laws.
Mitigation strategies
- Adoption of 'Poisoning' tools (e.g., Nightshade, ShieldFont) to protect personal images and text
- Legislation establishing 'Data Provenance' rights
- Mandatory 'Do Not Train' flags in web standards
- Deployment of Local-First AI models (on-device processing)
- Development of Differentially Private AI architectures (e.g., DP-RAG)
Top evidence
FTC Finalizes Orders with Cox Media Group, Two Other Firms Settling Charges They Deceived Customers About “Active Listening” AI-Powered Marketing Service
FTC Press Releases
The FTC has finalized orders against Cox Media Group and two other firms for deceiving customers about an 'active listening' AI-powered marketing service. These companies falsely claimed to target ads based on conversations captured from smart devices, highlighting a clear incident of privacy erosion and deceptive practices around AI capabilities.
When AI Takes Notes: Court Allows Privacy Claims Against Otter.ai to Proceed
AI Incident Database
Otter.ai's AI notetaker tool is facing a lawsuit for allegedly violating Illinois's biometric privacy law, with a California court allowing the privacy claims to proceed. This incident highlights the legal challenges and privacy concerns associated with AI systems collecting sensitive personal data.
China's Z.ai disables AI coding assistant features after security issue
AI Incident Database
Chinese startup Z.ai disabled features of its AI coding assistant after users reported it was uploading local code repositories to overseas cloud servers without permission. This incident highlights critical AI model integrity and supply chain vulnerabilities, where AI tools can inadvertently or maliciously exfiltrate sensitive data.
📱 Hey Siri, How Do I Limit AI Data Access? | EFFector 38.17
Electronic Frontier Foundation
This article from the EFF discusses the privacy implications of new AI features, such as those integrated into voice assistants like Siri on iOS 27. It raises concerns about how these AI tools handle user data and the challenges users face in limiting AI data access.
How to Limit What Apple’s New Siri AI Can Access in iOS 27
Electronic Frontier Foundation
Apple's new Siri AI in iOS 27, functioning as a chatbot, introduces new privacy concerns due to its expanded access to user data. The article provides users with practical steps to limit Siri AI's access, empowering them to maintain control over their personal information and digital privacy.