Photo by Gabriele Malaspina on Unsplash

Last month, a lawyer in New York filed legal briefs citing cases that don't exist. Not because he was careless, but because ChatGPT confidently invented them. The model had done what it does best: string together words that sound authoritative, grammatically correct, and utterly fabricated. This wasn't a glitch. This was the system working exactly as designed.

This phenomenon—where AI generates false information with complete confidence—is called hallucination. But that name is misleading. The AI isn't hallucinating in the way a human does. It's not confused or seeing things. It's following its core instruction: predict the next word based on patterns in training data. Sometimes that prediction leads to facts. Sometimes it leads to elaborate fictions presented as certainties.

How Hallucinations Happen (It's Simpler Than You'd Think)

Understanding AI hallucinations requires abandoning the idea that these models "know" things. They don't. What they do is assign probability to word sequences. When you ask a language model for a statistic, it's essentially playing an elaborate guessing game based on billions of examples it learned from.

Think of it this way: if your training data contains the phrase "studies show that 73% of office workers prefer remote work" (even if that number is made up), the model learns that this pattern exists. Later, when someone asks "what percentage of workers prefer remote work?", the model might reproduce something similar—or it might interpolate, creating a plausible-sounding variation like "approximately 71% of professionals." The model has no mechanism to fact-check itself. It only knows whether a sequence "looks like" something it has seen before.

The scariest part? Hallucinations often sound more confident than accurate information. A recent study from Stanford found that language models produce false information with near-identical confidence levels as true information. Your brain interprets confidence as reliability. The AI doesn't even know the difference.

The Numbers Game: Why Statistics Are Particularly Vulnerable

Numbers are uniquely susceptible to hallucination because there are so many plausible variations. If an AI model needs to generate a statistic, any number in a reasonable range sounds coherent. Is it 42%? Could be. 67%? Sure. 89%? That works too.

Consider what happened when a financial analyst used GPT-4 to research market predictions. The model generated three different statistics about semiconductor manufacturing, all sounding authoritative, all completely fabricated. The analyst didn't catch it until he tried to verify the sources. How many times does this happen and nobody notices?

Numbers also benefit from what researchers call "semantic drift." A model might learn that "statistics show X" is a common pattern in training data, then generate something statistically plausible even though it's false. It's pattern-matching without understanding. The model can't access the internet, consult databases, or verify anything. It's just predicting based on what it has seen.

Why We Fall For It (And What That Reveals About Us)

Here's the uncomfortable truth: we fall for AI hallucinations because we're primed to trust confident-sounding information, especially when it's detailed and specific. A vague answer sounds uncertain. A precise statistic with a source attribution (even a fake one) sounds researched.

Psychologists call this the illusory truth effect. Repetition and confidence make false information more believable. The AI is an unwitting master of this. It presents fabrications with such grammatical polish and structural coherence that our brains accept them. We're wired to trust authoritative-sounding language, and the AI has learned to produce exactly that.

There's also an element of trust transference happening. If ChatGPT got the first answer right, we assume it'll get the second one right too. This is how the lawyer ended up citing fake cases. He had successfully used the AI for initial research. Confirmation bias made him lower his guard.

The Real Problem: We're Building Institutions Around Uncertainty

The hallucination problem becomes critical when institutions start relying on these tools without proper verification protocols. A marketing team using AI to generate "statistics" for a campaign doesn't seem dangerous until those statistics end up in a pitch to investors. A researcher using AI to speed up literature reviews might miss that three citations don't actually exist.

What makes this particularly tricky is that the AI world is still treating hallucination as a technical problem with a technical solution. Engineers are working on better prompting, improved training data, and retrieval-augmented generation (feeding the model real information sources). These help, but they don't solve the fundamental issue: prediction-based systems will always have a margin of error, and that margin includes invented information.

Some researchers now recommend treating AI output more like human brainstorming—useful for ideas and direction, but requiring verification before deployment. Others advocate for transparency requirements, where AI-generated content is clearly labeled. A few are pushing for mandatory fact-checking pipelines.

The uncomfortable reality is that AI hallucinations are a feature of how these systems work, not a bug that will be fixed. We're not going to engineer our way out of this problem completely. We're going to have to engineer around it, building human verification into our workflows and remaining skeptical of confident-sounding information that appears suspiciously detailed.

The lawyer's fake cases are a symptom of a larger problem: we're rapidly integrating AI into critical decision-making processes before we've figured out how to integrate verification. That's the real hallucination—our hallucination that we can trust the output without checking.

For a deeper exploration of AI's other confidence problems, check out our article on how AI learned to fake expertise and the confidence crisis nobody's talking about.