How personal do you get with your chatbot? Does it interpret your lab results? Help you sort out your finances? Offer advice at 2 a.m. when your worries are particularly existential? Without thinking about it too deeply, you might be revealing a whole trove of personal information about yourself, and that could be a problem. At a time when people are increasingly integrating chatbots into their everyday lives, researchers are trying to work out the implications of feeding AI personal information.
Consider this startling statistic: 43% of workers say they’ve shared sensitive information with AI – including financial and client data. Just over half of US adults use large language models, according to a 2025 study from Elon University. What’s more, chatbots are designed to be friendly and keep people chatting – and talking about themselves.
The Unpredictable Journey of Your Data
“The ultimate problem is that you just can’t control where the information goes, and it could leak out in ways that you just don’t anticipate,” said Jennifer King, privacy and data policy fellow at Stanford Institute for Human-Centered Artificial Intelligence. As abstract as that theory may sound, researchers like King say it’s worth considering exactly what you’re telling chatbots, and what repercussions that info might have in the future.
One question researchers have is whether models memorize information and, if so, whether that information can be coaxed back out verbatim or near-verbatim. Memorization is actually one of the core complaints in The New York Times’ lawsuit against OpenAI. (OpenAI, in a statement from 2024, said “regurgitation is a rare bug” it’s trying to eliminate.)
From Personal Secrets to Surveillance Concerns
A concern is that all of this data might be used for surveillance, King said. If that fear sounds alarmist, King called back to Anthropic’s tussle with the Department of Defense in the last few weeks, where the company objected to its product being used for mass domestic surveillance. “One of the most important things that came out of that was the kind of tacit admission that these things can be used for mass public surveillance,” she said. “This is exactly the type of thing that we would be worried about, that you can use these models to look across so many different data points.”
And even if models don’t have specific data, they might still be able to make predictions about people. In a piece for Stanford about her team’s research, King gave the example of a request for heart-healthy dinner ideas getting filtered through a developer’s ecosystem, classifying you as a “health-vulnerable” person, and that info ending up in the hands of an insurance company.
The Business Reality: AI Companies Under Scrutiny
This surveillance concern isn’t just theoretical – it’s playing out in real-time legal battles. Anthropic submitted sworn declarations to a California federal court on March 21, 2026, contesting the Pentagon’s designation of the company as a national security risk. The declarations, filed by Sarah Heck (Head of Policy) and Thiyagu Ramasamy (Head of Public Sector), argue that the government’s claims rely on technical misunderstandings and issues never raised during negotiations.
Heck revealed that on March 4, Pentagon Under Secretary Emil Michael emailed CEO Dario Amodei stating the two sides were ‘very close’ on the very issues now cited as security threats (autonomous weapons and mass surveillance), contradicting later public statements. Ramasamy explained that once Anthropic’s Claude AI is deployed in air-gapped government systems, the company has no access or ability to interfere, and employees undergo U.S. security clearance vetting.
User Concerns vs. Business Priorities
While companies navigate these complex regulatory waters, users are grappling with their own concerns. A global survey of over 80,000 Anthropic Claude chatbot users across 159 countries reveals that AI hallucinations (mistakes) are the top concern (27%), surpassing job displacement (22%). The study, conducted in 70 languages, found that 32% of users reported increased productivity, with regional differences showing more optimism in lower and middle-income countries.
Deep Ganguli, who leads Anthropic’s societal impacts team, noted the importance of this user feedback: “collect this rich human experience using Claude, so it could really inform our research agenda, change our research agenda, change the way we think about building our products, deploying our products.”
Practical Steps for Businesses and Individuals
For businesses looking to implement AI responsibly, Joel Hron, CTO at Thomson Reuters Labs, offers practical advice: “We’re not in the 90% game. We’re in the 99% and 99.9% game, and we must consider how we get that extra nine or two nines of accuracy, which is the difference for trust.” Thomson Reuters has taken this approach seriously, launching the Trust in AI Alliance with Anthropic, AWS, Google Cloud, and OpenAI.
For individual users, King recommends several practical steps:
- Understand your platform settings – some chatbots offer private chats that aren’t saved to history or used for training
- Delete old conversations and personalizations
- Remember whether you’re using a personal or work account (“There’s no employee expectation of privacy there”)
- Be aware that humans might be reading your messages for reinforcement learning
The Regulatory Gap and Future Implications
What makes any of these points especially tricky is the lack of regulation around how AI companies store sensitive data. The California Consumer Privacy Act, for example, has certain requirements around how data like medical records need to be treated differently from other forms of data. But regulation in the US may differ from state to state, and at the federal level – well, there is no regulation.
“If we had the law that protected us, it wouldn’t be so much of a risk,” King said. As AI continues to evolve from a novelty to an essential tool, the tension between innovation, privacy, and security will only intensify. The question isn’t whether we’ll use AI – we already are – but how we’ll navigate the complex web of trust, transparency, and regulation that comes with sharing our most sensitive information with machines designed to be our confidants.