Photo by Gabriele Malaspina on Unsplash

Last Tuesday, I watched a large language model confidently explain why the mitochondria is not, in fact, the powerhouse of the cell. It wasn't joking. The model had constructed an elaborate, internally consistent argument that sounded plausible until you remembered high school biology. This wasn't a glitch or a rare failure—it was the model doing exactly what it was designed to do: generate the next statistically likely word, regardless of whether that word was true.

This phenomenon, known as epistemic overconfidence, has become one of the most important problems in AI safety and reliability. And if you're planning to deploy AI systems in any meaningful way, you need to understand it.

The Confidence Problem Runs Deeper Than You'd Think

Here's the uncomfortable truth: modern AI systems have no built-in mechanism for uncertainty. When a neural network produces output, it assigns probability scores to each token it generates, but these numbers don't actually map to "how sure am I." They map to "which prediction matches my training data best." These are profoundly different things.

A 2023 study from UC Berkeley analyzed how language models performed on questions where they claimed certainty. Researchers discovered that when models expressed high confidence (using phrases like "definitely" or "certainly"), they were only correct about 70% of the time. When they expressed low confidence, they were right roughly 72% of the time. The confidence signals were almost completely decoupled from actual accuracy.

Think about that for a second. The model's subjective confidence level is essentially a coin flip when it comes to predicting correctness. Yet users consistently interpret high-confidence responses as reliable information. A doctor using AI to suggest diagnoses. A lawyer using it to research precedents. A financial advisor using it for market analysis. Each one might be interacting with a system that's essentially performing a sophisticated version of guessing.

Why AI Systems Became Overconfident Bullshitters

The path to this problem is almost poetic. Large language models are trained using something called next-token prediction. The model sees a sequence of words and learns to predict which word should come next. Repeat this millions of times across billions of tokens of text, and you get something that's eerily good at pattern matching.

But here's the critical part: the training process optimizes for one thing only—reducing prediction error. It doesn't optimize for truthfulness, calibration, or epistemic humility. If a confident-sounding hallucination is statistically likely given the input, the model will generate it with the same probability as it generates truth.

This gets worse when you factor in human feedback. Systems like ChatGPT get fine-tuned using something called RLHF (reinforcement learning from human feedback). Raters score different model outputs and feed that back into the training. The problem? Humans often rate confident, well-structured responses as higher quality—even when they're wrong. A clear, persuasive explanation of incorrect information gets rated higher than an accurate but hedged response. The system learns that confidence is rewarded.

So we've essentially trained AI systems to be extremely confident liars. Or rather, we've trained them to be confident pattern-matchers that happen to be wrong sometimes. The distinction is important, because it means this isn't solvable by just feeding the model more truthful training data. The fundamental architecture doesn't distinguish between truth and plausible falsehood.

What Actually Happens When AI Hallucinates

Last month, a lawyer in New York submitted a brief citing six non-existent court cases. The citations were perfect. They had the right formatting, the right type of citations, the right density of legal jargon. Everything about them screamed "real case law." The AI system that generated them wasn't trying to deceive. It was doing what large language models do: generating text that statistically matches legal briefs.

This connects to a broader phenomenon worth understanding. When AI systems "hallucinate" (generate false information), they're not making random errors. They're typically making errors that fit the expected pattern of their training data. A legal AI generates fake court cases that look like real cases. A medical AI might hallucinate symptoms that are statistically correlated with other symptoms. A research AI will fabricate citations that match the format and topic of real citations.

These aren't random failures. They're systematic failures that emerge from the architecture itself. And because the system has no independent way to verify its outputs, it can't catch them.

The Uncomfortable Solution (It's Not Fancy)

The most effective current approach to this problem is also the least sexy: human verification. For high-stakes applications, you need someone with real expertise to check the AI's work. Not to use it as a starting point—to actually verify it.

Some organizations have started implementing this more formally. OpenAI's system cards now explicitly note which use cases require human oversight. The FDA has started requiring human verification for AI systems used in medical diagnostics. Healthcare institutions that deployed AI most effectively typically had doctors reviewing every recommendation.

This feels like admitting defeat—you're paying humans to check a machine that was supposed to save you money. But it's actually the realistic middle path. AI systems are phenomenal at certain specific tasks. They're excellent at drafting, brainstorming, explaining complex topics, and generating starting points for human work. They're terrible at being the final authority on anything that matters.

What's changed is that we now understand why they're overconfident. It's not a bug to be patched. It's a fundamental property of how they work. Understanding that changes how you should deploy them. You stop treating them as oracles. You start treating them as extremely convincing colleagues who need supervision.

If you want to understand more about the gap between what AI appears to know and what it actually understands, check out our piece on confident incompetence in machine learning—it explores the philosophical underpinnings of this problem.

The Real Question Going Forward

We've stopped asking "can AI systems do this task?" and started asking "how much do we have to supervise AI to make it safe for this task?" That's progress. The systems aren't going to become honest or humble on their own. But we can build processes and workflows around them that acknowledge their limitations.

The confidence problem won't disappear. But ignoring it—or worse, trusting it—will definitely cause problems. Make sure you're not one of those organizations that learns this lesson the hard way.