Photo by Microsoft Copilot on Unsplash

Last year, a researcher discovered that OpenAI's GPT models could reproduce verbatim passages from copyrighted books during conversations. Not paraphrases. Not summaries. Exact excerpts, complete with typos from the original editions. The AI hadn't been programmed to memorize anything—it had simply absorbed millions of texts during training and retained fragments like a sponge that never forgets.

This wasn't a bug. It was a feature of how these systems work. And it raises a question that keeps privacy advocates up at night: what else are these models remembering that we don't know about?

The Training Data Problem Nobody Wants to Talk About

Here's the uncomfortable truth: we don't actually know what's in most AI training datasets. Companies like OpenAI, Google, and Meta have been deliberately vague about their sources. They cite "publicly available data" and move on. But "publicly available" is doing a lot of work in that sentence.

A 2021 study by researchers at University of Washington found that Common Crawl—one of the largest datasets used to train modern AI systems—includes medical records, financial data, and personally identifiable information harvested directly from the web. Not anonymized. Not redacted. Real people's real secrets, now encoded into the weights and parameters of machine learning models.

One researcher scraped the training data of a popular AI image generator and found photographs of people, including some taken in private spaces. The images were credited to photographers under Creative Commons licenses, but the people photographed? Nobody asked their permission.

The scale of this is staggering. GPT-3 was trained on 570 gigabytes of text. That's roughly equivalent to 400 billion words. No human could possibly review all of that to ensure nothing sensitive made it through. The process is essentially:

1. Point a web scraper at the internet

2. Download everything

3. Train a model on it

4. Hope nothing goes catastrophically wrong

Spoiler alert: sometimes things go catastrophically wrong.

When Your Private Email Becomes AI Training Data

Consider what happens when your email is breached. Thousands of companies have been compromised over the past decade, exposing everything from passwords to intimate correspondence. Some of that data ends up in public repositories or dark web marketplaces. Some gets indexed by search engines. And some of it eventually gets scraped by someone building a dataset to train an AI model.

There's a disturbing case from 2022: researchers found that a language model had memorized and could reproduce sensitive information from a leaked database. When prompted correctly, it would reveal details from people's private communications. The model wasn't trying to be malicious. It had simply learned patterns from the data it was fed.

This is particularly concerning for vulnerable populations. Therapy transcripts, medical consultations, abuse survivor testimonies—all potentially part of some dataset somewhere, now baked into an AI system that anyone can interact with.

The Extraction Problem: Getting Your Data Back Out

Here's where it gets really weird: once information is absorbed into a neural network, you can't just ask it to "forget." It's not stored as a database entry you can delete. It's diffused throughout billions of parameters, woven into the mathematical fabric of the model itself.

Companies have started talking about "machine unlearning"—techniques to remove specific information from trained models. But it's still experimental. A 2023 paper from researchers at MIT showed that even after applying unlearning techniques, models could still partially reconstruct the original data under certain conditions.

This matters because unlike traditional databases, you have virtually no legal right to have your information removed from an AI model. The General Data Protection Regulation (GDPR) in Europe has a "right to be forgotten," but enforcement against AI companies has been weak. And in the United States? There's barely any regulation at all.

You can ask Google to remove your information from search results. You can file DMCA takedowns to prevent your work from being used in certain contexts. But there's no equivalent mechanism for AI training data.

What Actually Gets Memorized (And Why It's Unpredictable)

The frustrating part is that model memorization is essentially random from a user's perspective. A model might faithfully reproduce someone's credit card number from training data, or it might forget a famous quote it saw a thousand times. There's no consistent pattern that lets us predict what will be remembered.

This unpredictability creates enormous liability. If an AI model reproduces someone's private information in front of a witness, the company faces potential legal action. But they can't audit the model to check for sensitive data without essentially running every possible prompt—which would take centuries.

Some companies are now using "differential privacy" techniques, which add mathematical noise to training data to reduce memorization. But this comes with a cost: models trained with differential privacy are often less capable and less accurate.

It's a genuine dilemma. You can have a more powerful model that might leak private information, or a weaker model that's safer. For companies optimizing for performance, the choice is obvious.

What Happens Next?

The situation is starting to shift, albeit slowly. Lawsuits are piling up. In 2023, authors including Sarah Silverman filed suit against OpenAI and Meta, alleging copyright infringement through unauthorized use of copyrighted books in training data. Similar cases are emerging in Europe.

At the same time, companies are getting more cautious about their training data sources. Some are now licensing data explicitly. Others are building models from "cleaner" sources like employee-written content.

But without serious regulation—the kind that treats AI training data the way we treat medical records or financial information—the fundamental problem persists. Your secrets are out there, encoded into systems you'll never fully understand, controlled by companies with every incentive to keep improving their models and none to prioritize your privacy.

For more insight into how these systems behave when they have access to sensitive information, check out Why Your AI Chatbot Keeps Saying Confidently Wrong Things (And How to Fix It).