Photo by Microsoft Copilot on Unsplash

Last month, a lawyer in New York submitted a court filing citing six cases that didn't exist. He'd asked ChatGPT to find relevant precedents, and the AI obliged—complete with fake case names, dates, and judicial citations. The judge was not amused. But here's what's really unsettling: ChatGPT didn't accidentally confuse real cases. It invented them from scratch and presented them with absolute certainty.

This phenomenon isn't a bug. It's baked into how these systems fundamentally work.

Why AI Systems Can't Tell the Difference Between Truth and Fiction

Most people assume AI hallucinations happen because the model ran out of training data or encountered something unfamiliar. Wrong. The real issue is far more interesting and troubling.

Modern language models like GPT-4 or Claude don't have a built-in concept of "truth." They're trained using something called next-token prediction. Imagine you're playing a word-completion game where the goal is simply to predict the most statistically likely next word based on patterns in billions of text examples. That's essentially what's happening under the hood.

When you ask an AI a question, it's not searching a database of facts. It's generating text one word at a time, always choosing what seems most probable given everything that came before. If your question is "What is the capital of France?" the model has learned from massive amounts of training data that France→Paris is a very common pattern. Easy answer.

But ask it something more obscure—like "What papers did researcher Susan Chen publish in 1997?"—and the model faces a problem. It doesn't actually "know" whether Susan Chen exists or what she published. But it has learned patterns about how academic citations are structured. So it generates something that sounds academically plausible. A name. A year. A journal title that sounds real.

The model isn't thinking, "I should make this up." It's doing what it was designed to do: generate the next statistically likely token. And because academic citations follow predictable patterns, the output looks absolutely convincing. Even to people who know the field.

The Confidence Problem: Why AI Lies Without Hesitation

Here's what keeps security researchers up at night: AI systems don't express uncertainty. They don't say "I'm not sure" or "I might be wrong here." They commit to fabrications with the same tone and confidence they use for established facts.

This isn't accidental. During training, AI systems are rewarded for providing complete, coherent responses. Saying "I don't know" is punished because it's considered an incomplete answer. The system learns to always finish the task, always provide an answer, always sound certain.

A researcher at Stanford tested this by asking ChatGPT about a completely made-up mathematical concept. The AI didn't just fabricate an explanation. It provided a detailed, multi-paragraph response that sounded like it came from a legitimate textbook. When pressed for sources, it invented citations. When asked to verify its claims, it double down on the false information.

This behavior isn't stupidity. It's a direct result of optimization. The model was trained to be helpful, harmless, and honest—but when these goals conflict, "helpful" (providing an answer) often wins.

Why This Gets Worse Before It Gets Better

The uncomfortable truth is that as AI systems get more powerful, this problem likely gets worse. Bigger models trained on more data become better at generating plausible-sounding text. They learn more sophisticated patterns about how to structure false information so it passes scrutiny.

Imagine a future AI trained on not just text, but also video, audio, and interactive media. The patterns it learns about how information is presented become even more refined. The hallucinations become even more convincing.

We're also seeing a troubling arms race. As companies add safeguards to prevent hallucinations, users find workarounds. They ask follow-up questions in ways that trick the model into committing to false information. They use jailbreaks that disable safety mechanisms. The system learns to be even more confident in its responses.

One particularly worrying study found that when multiple users ask an AI the same question, and the AI generates different answers, the AI will often commit to whichever answer is most popular—regardless of accuracy. It's learning to follow crowd consensus rather than truth.

What Actually Happens Inside the Model

To understand why this happens, you need to understand what these systems literally cannot do: they cannot access external information in real-time. They cannot browse the internet. They cannot check a database. Everything they say comes from patterns encoded in their weights—essentially billions of mathematical parameters learned during training.

When you ask an AI about something specific, there's no mechanism for it to say "I need to verify this before responding." There's no pause for fact-checking. The weights were set during training and they don't change when you interact with the model.

This is why some researchers are exploring retrieval-augmented generation (RAG), where AI systems are connected to actual databases and search tools. Instead of relying purely on learned patterns, they pull real information and incorporate it into responses. But even this approach has limitations.

For a deeper look at why this problem keeps getting worse as AI systems advance, check out Why AI Hallucinations Are About to Get Exponentially Worse (And What That Means for You).

What Users Should Actually Do Right Now

The practical answer is unsexy: don't treat AI outputs as ground truth for anything important. Use AI for brainstorming, drafting, and exploration. Use it to accelerate thinking. But verify everything that matters.

This isn't a temporary limitation that will disappear with the next version. It's a structural feature of how these systems work. Until AI can access real-time information and can genuinely express uncertainty, hallucinations will remain a core issue.

The real question isn't "when will AI stop hallucinating?" It's "how do we build systems that can acknowledge what they don't know?" And that's a problem that requires different architecture, different training approaches, and fundamentally different assumptions about what AI should do.

Until then, healthy skepticism isn't just reasonable. It's essential.