Photo by Igor Omilaev on Unsplash

Last month, a researcher submitted a paper to a peer-reviewed journal that cited a groundbreaking 2019 study on neural plasticity. The study sounded plausible. The authors' names made sense. The journal name was real. There was just one problem: the paper didn't exist. The researcher had asked ChatGPT to help write the literature review, and the AI had simply fabricated the entire citation.

This isn't an isolated incident. It's becoming a pattern. And it's revealing something deeply troubling about how we've built these increasingly powerful AI systems: they're fundamentally broken at distinguishing between what they've actually learned and what they're inventing.

The Hallucination Problem Is Getting Worse, Not Better

"Hallucination" is the polite term researchers use when AI models generate confident but completely false information. But polite language hides a serious crisis. When OpenAI released GPT-4, many people assumed the company had solved this problem. The model was more sophisticated, trained on more data, and produced fewer obvious errors. Yet internal testing revealed something unexpected: GPT-4 actually hallucinated more frequently than GPT-3.5 in certain scenarios.

Why? Because as models get larger and more capable at pattern-matching, they become better at sounding confident about things they shouldn't be confident about. They've learned to mimic the statistical patterns of authoritative writing so well that they can write convincing fiction without hesitation.

Consider what happened when researchers at Stanford asked ChatGPT to cite papers on vaccine side effects. The AI produced citations with author names, publication years, and journal names—all formatted correctly. Approximately 80% of them were completely made up. Not slightly misrepresented or paraphrased. Entirely fabricated. Yet the formatting was so perfect that a casual reader would have no reason to doubt them.

Why This Matters More Than You Think

You might assume this problem is confined to academics who are careless enough to trust an AI. That's a comforting thought, but it's wrong. This issue is already contaminating the internet at scale.

Medical professionals are beginning to use AI tools to quickly summarize research. Legal firms are experimenting with AI for due diligence. Journalists are using ChatGPT for background research. Each of these professions operates on the assumption that citations point to real sources. When they don't, the entire foundation crumbles.

There's also a darker feedback loop forming. As more people use AI-generated content online, that content gets indexed by search engines and potentially used to train future AI models. So hallucinated citations could actually become "training data" for the next generation of AI, which would then hallucinate with even greater confidence.

A particularly unsettling example: someone asked Claude (Anthropic's AI) to write a historical account of a fictional event. The AI didn't just make up the event—it fabricated quotes from historical figures, invented plausible-sounding book citations, and even created fictional Wikipedia articles that supposedly documented the event. The response was internally consistent enough that someone skimming it might accept the entire narrative as fact.

The Technical Root of the Problem

Large language models work by predicting the next word based on patterns in training data. They're not databases. They don't retrieve information. They statistically generate text that "feels right" given the context. When you ask them to cite a paper, they're not looking it up—they're predicting what a citation should look like based on billions of examples of real citations.

This creates a fundamental mismatch between what humans expect and what the AI actually does. We ask it a factual question and interpret its response as a retrieval of information. The AI is actually just making an educated guess about what kind of text should come next.

The scariest part? The model has no way to distinguish between "I saw this pattern 10,000 times in my training data" and "I'm making this up right now." It can't inherently know the difference. Researchers have tried adding confidence scores, disclaimer generation, and retrieval systems that pull from verified databases. Some of these help, but none are foolproof.

This connects to a broader issue we've explored before: why your AI chatbot becomes dumber when you ask it the right questions. The same architectural limitations that make models fail under scrutiny also make them prone to confident confabulation.

What We Need to Do Now

This isn't a problem that will solve itself. We can't wait for better models because better models might just get better at sounding confident about false things.

First, institutions need to adopt verification protocols immediately. If you're using AI to generate citations, you must check every single one. This sounds tedious, but it's the only honest approach right now. Some universities are updating their academic integrity policies to explicitly ban citations generated by AI without verification.

Second, AI companies need to be transparent about limitations. Anthropic and OpenAI have both published research acknowledging the hallucination problem, which is good. But they also need to be more aggressive about disabling citation generation when they can't verify sources, even if it means the AI says "I don't know" more often.

Third, we might need new standards for AI-generated content. Some researchers are proposing that AI-written material include metadata about which statements were verified, which came from training data with confidence scores, and which are speculative. It sounds bureaucratic, but it could prevent countless errors.

The uncomfortable truth is this: we've built systems that are remarkably good at sounding like they know what they're talking about. That's a feature in some contexts and a catastrophic bug in others. Until we figure out how to tell the difference, we should approach any AI-generated citations—any AI-generated factual claims—with a skepticism that feels paranoid but is actually just appropriate caution.