Photo by Gabriele Malaspina on Unsplash

Last week, I asked ChatGPT who won the 1987 World Series. It told me the Minnesota Twins did. Confidently. Definitively. No hedging whatsoever. The problem? The Minnesota Twins won in 1991. The 1987 champion was also the Minnesota Twins—but that's beside the point. What mattered was that the AI didn't say "I'm not sure" or "I don't have reliable data on that." It just made something up and served it with absolute conviction.

This phenomenon has a name in AI research circles: hallucination. But that clinical term doesn't capture what's actually happening. Hallucination implies the model is dreaming, seeing things that aren't there. The truth is messier and more troubling. These language models aren't confused. They're working exactly as designed—just in ways we don't fully understand.

The Architecture of Confidence Without Knowledge

Here's the weird part about how large language models work: they don't actually know anything. They're probability machines dressed up in conversational clothes. When you ask a question, the model doesn't retrieve a fact from some internal database. Instead, it predicts the statistically most likely next sequence of words, then the next, then the next.

Think of it like this. You're standing in a train station where the next word is always predicted by what came before it. The model learned patterns from billions of sentences during training, so it got really good at predicting what word typically follows other words. But predicting probable text isn't the same as understanding facts. A model trained on the entire internet learned that sentences about the 1987 World Series usually contain words like "Minnesota" and "Twins" and "won." So when you ask about that World Series, it strings those words together in a grammatically sound way because that's what it does—it optimizes for sounding right, not for being right.

The trouble starts because this prediction process is invisible. You see fluent English. You hear authority in the phrasing. Your brain interprets the confidence as expertise. But there's nothing behind the curtain except statistics and matrix multiplication.

When Probability Masquerades as Certainty

A researcher at Stanford ran an experiment where they asked GPT-3 and GPT-4 to answer questions where the correct answer contradicts common assumptions. For instance: "A bat and ball cost $1.10 total. The bat costs $1 more than the ball. How much does the ball cost?" Most people instinctively say $0.10, but the correct answer is $0.05.

GPT-4 got that right. But when researchers asked it variants where the correct answer contradicted facts the model saw frequently during training, accuracy tanked. The model didn't switch to saying "I don't know." It confidently generated plausible-sounding but incorrect answers.

What's happening here is that the model has learned to optimize for producing text that sounds authoritative and complete. Saying "I'm not sure" gets a lower probability score in the training data than generating a full answer. During training, the model was rewarded (through the loss function) for producing text that matched human-written responses. Those human responses rarely include genuine uncertainty. So the model learned: always finish your thought. Always provide an answer. Doubt is for sissies.

This is particularly insidious in domains where answers matter. Medical queries. Legal questions. Financial advice. A model might confidently recommend a medication interaction that doesn't exist, or cite a legal precedent that was never decided, because those wrong answers still match the statistical patterns the model learned.

The Illusion of Understanding

You might think that better training data would solve this. Just feed the model only true information, right? The problem is far stickier. The model doesn't actually have access to a reliable truth database at runtime. It's still just predicting the next word. And during its training phase, it saw billions of examples of human-written text that included mistakes, urban legends, hoaxes, and outright fabrications mixed in with accurate information.

There's also something called the "gating problem." Even if a model somehow knew whether its generated answer was likely to be accurate, it wouldn't have an obvious way to stop itself from generating false information. When AI Becomes Your Unreliable Expert: How Language Models Convinced Us They Understand What They're Actually Guessing explores exactly this issue—these systems have learned to sound authoritative in ways that bypass our skepticism.

Some researchers are trying to build in uncertainty estimation, teaching models to generate probability scores alongside answers. "I'm 87% confident the answer is X." But even this approach has limits. A model saying "I'm 95% confident" doesn't mean it actually performed statistical analysis. It learned the phrase from training data where humans expressed confidence, and it's pattern-matching to that.

How to Actually Use These Tools Without Getting Burned

This doesn't mean language models are worthless. They're genuinely useful for brainstorming, drafting, coding assistance, and many other tasks. But using them safely requires understanding their actual nature.

First, treat factual claims as starting points, not conclusions. Ask a language model for draft copy or creative ideas, but verify any specific facts independently. This sounds obvious, but most people don't do it. They trust the confidence in the model's tone.

Second, ask follow-up questions. Ask the model to cite sources. Ask it to express uncertainty. These prompts can sometimes—not always—make it more careful. A model prompted to "think step-by-step" performs better than one given a question cold, probably because the intermediate reasoning steps create natural checkpoints.

Third, use models that have been trained with explicit techniques to reduce hallucination, like retrieval-augmented generation, where the model can look up information from a trusted database before generating answers.

Most importantly, stay skeptical. The fluency is real. The confidence is a natural byproduct of how these systems work. But fluent and confident doesn't mean accurate. Not even close.