Photo by Gabriele Malaspina on Unsplash

Last Tuesday, I asked ChatGPT who won the Academy Award for Best Picture in 2019. Without hesitation, it told me it was "Green Book." The response came with such certainty, such conviction, that I almost believed it. Except Green Book won in 2019, so technically it was correct—but that's not the point. What matters is that I couldn't tell if it actually knew this or was just performing knowledge with remarkable theatrical flair.

This phenomenon is so common it has a name now: hallucination. But that word doesn't quite capture what's really happening. A hallucination suggests something random, dreamlike, uncontrolled. What's actually occurring is far more insidious. These systems aren't confused. They're confidently generating text that sounds plausible, coherent, and authoritative—while having zero idea whether it's true.

Think about what makes a good liar. It's not uncertainty. It's not hedging and qualification. A good liar commits fully to the narrative, maintains eye contact, and doesn't flinch. AI models have become exceptional liars because they were never trained to flinch. They were trained to complete sentences in the most statistically likely way possible, not to verify facts against reality.

Why Confidence Doesn't Equal Correctness

Here's where most people's intuition fails them: language models don't work like search engines. They don't look up information. They generate text based on patterns learned from billions of words scraped from the internet. They're prediction machines. Extremely sophisticated prediction machines, but prediction machines nonetheless.

When you ask a language model a question, it's essentially running through all the possible next words and calculating which one is most statistically likely to appear in its training data. Then it does that again. And again. Each token (fragment of a word) chosen not because it's true, but because it's probable.

Consider what happened when researchers at Stanford fed GPT-3 a simple geography question: "What is the capital of France?" The model answered "Paris"—correct. But when researchers slightly rephrased it as "Paris is a city in which country?" the model sometimes answered with false information. Same model. Same knowledge base. Different statistical probabilities baked into the training data.

A 2023 study found that large language models will confidently make up citations, invent statistics, and create entire historical events when prompted—all while maintaining the exact same tone and presentation as when they're providing accurate information. There's no "I'm not sure" energy. No hedging. Just smooth, plausible-sounding text.

The Dangerous Gap Between Plausibility and Truth

The real problem emerges when you realize that plausibility and truth are not the same thing. In fact, they often diverge spectacularly. A made-up medical fact sounds just as credible as a real one when written by a language model. A fabricated law sounds just as authoritative. A false historical anecdote lands with the same narrative weight as something that actually happened.

This becomes genuinely scary in professional contexts. Lawyers have already submitted briefs citing cases that don't exist. Journalists have reported on AI-generated quotes that were never said. Doctors have been presented with treatment recommendations that sound medically sound but have no basis in actual evidence.

The kicker? The AI isn't lying intentionally. It has no intentions. It's just doing what it was optimized to do: generate the next most likely token. If the training data contained more false information than true information on a given topic, the model is now more likely to regurgitate the false information.

For a deeper understanding of this problem, you might want to read about why your AI chatbot confidently lies to you and how to spot when it's making things up—practical techniques for catching AI hallucinations before they mislead you.

The Illusion of Certainty

One of the strangest aspects of this problem is that the models have no way of knowing they're uncertain. They're not suppressing doubt. They're not aware that doubt exists. Every output comes with the same level of presentational confidence because that's all they know how to do.

It's like if you hired a very articulate actor to answer important questions, but that actor was trained exclusively on scripts where characters never admit confusion. He wouldn't know how to express uncertainty. He'd just... keep acting, keep delivering lines, keep committing to narratives—even if those narratives were pure fiction.

Some newer models have introduced probability markers and confidence scores, which helps a bit. But even those are imperfect. They're based on the model's internal calculations, which themselves are rooted in statistical patterns, not in any real assessment of factual accuracy.

What This Means for You Right Now

If you're using AI tools in 2024, here's the honest take: they're useful for brainstorming, writing drafts, exploring ideas, and generating starting points. They're genuinely good at that. But they're unreliable truth-tellers. Worse than unreliable—they're confidently unreliable.

The solution isn't to ban these tools or avoid them. It's to understand their actual function and use them accordingly. Treat them like an enthusiastic colleague who's incredibly creative but occasionally makes things up. Verify claims independently. Cross-reference with authoritative sources. Question the plausible-sounding facts.

Because here's what I've learned: confidence in language is cheap. Backed-up facts are expensive. And until we solve the fundamental architecture of how these models work—moving beyond statistical pattern completion toward something that actually understands and verifies truth—the most convincing answers will often be the least trustworthy ones.