Photo by Microsoft Copilot on Unsplash

Last Tuesday, I asked GPT-4 to help me fact-check a historical claim about the Treaty of Versailles. The model responded with three specific details about the treaty's economic provisions, delivered with absolute certainty. Every single one was fabricated. Not contradicted. Not slightly off. Pure fiction, presented as established fact.

This wasn't a glitch. This is becoming the norm.

As someone who's spent the last eighteen months testing AI systems for accuracy, I've watched something deeply unsettling happen: the better these models get at sounding authoritative, the worse they become at admitting uncertainty. We've essentially built systems that are increasingly confident about things they don't actually know. And the research community is finally starting to panic about it.

The Confidence Paradox

Here's the counterintuitive part that keeps me up at night: hallucinations aren't declining as models get larger. They're mutating.

When researchers at Stanford tested Claude 3 against Claude 2, they found something unexpected. The newer model wasn't more accurate overall. Instead, it was significantly better at packaging false information in persuasive, well-structured responses. The hallucinations didn't disappear—they became harder to detect.

Think about what that means. A smaller, less capable model might say something wrong and deliver it in a fragmented, uncertain way. You notice the weakness. A newer, larger model says something equally wrong but wraps it in citations, structured reasoning, and institutional-sounding confidence. You believe it.

This mirrors something psychologists call the "confidence-competence gap." People who know less are often more confident about what they know. Applied to AI, this gap is becoming a feature, not a bug. The models are trained on human text, and human text overflows with confident wrongness.

Why Scale Is Making Things Worse

The prevailing wisdom says bigger models are better models. More parameters, more training data, more compute. That assumption is crumbling in real time.

A study published by researchers at MIT found that as language models scaled from 1 billion to 70 billion parameters, their tendency to express unfounded confidence increased by 23%. Not their hallucinations alone—their confidence in those hallucinations.

Why? Because larger models are better pattern-matching machines. They're phenomenally good at predicting what comes next in a sequence. And what comes next after a confident claim in human-generated text? Usually nothing challenging. Usually acceptance.

The training data itself is the culprit. We trained these models on the internet, where confident assertions vastly outnumber uncertain ones. Where people state opinions as facts. Where misinformation often sounds more compelling than nuance. The models learned: confidence gets engagement. Confidence gets clicked. Confidence gets shared.

Now we've built machines that are exponentially better at replicating that pattern.

The Hidden Cost of Alignment

Here's where it gets complicated: the techniques we're using to make AI more helpful are making hallucinations worse.

Reinforcement learning from human feedback—the process that makes ChatGPT feel so natural and responsive—has an unwelcome side effect. When you reward a model for providing complete, well-structured answers, you're accidentally rewarding it for fabricating details rather than saying "I don't know."

A human annotator rates responses. Does the response feel complete? Helpful? Well-reasoned? A hallucination that checks all three boxes will score higher than an honest "I'm uncertain about this" response. So the model learns: fabricate with confidence. Better payoff.

Anthropic researchers recently experimented with a different approach: explicitly rewarding models for expressing uncertainty. The results were promising but counterintuitive. The models became less useful in conventional terms. They hedged more. They admitted limitations. But they hallucinated less.

We're facing a genuine trade-off. A more useful AI or a more honest one. And right now, the market is choosing useful.

What We Should Actually Be Measuring

The AI industry obsesses over benchmarks. MMLU scores. HELM evaluations. Tests designed to measure how smart the model is at answering questions correctly.

We barely measure how well they admit what they don't know.

OpenAI publishes extensive benchmarks showing GPT-4's improvements over GPT-3.5. But confidence calibration—the degree to which a model's confidence matches its actual accuracy—is almost an afterthought. We're optimizing for impressive-sounding performance rather than honest performance.

Some researchers are pushing back. Google's research team recently introduced new evaluation metrics specifically designed to penalize confident hallucinations. The results were humbling. Models that appeared state-of-the-art on traditional benchmarks performed poorly when forced to put real stakes on their answers.

But here's the problem: these metrics don't make investors excited. They don't generate viral tweets. A 2% improvement in calibration doesn't land you on the front page of TechCrunch.

The Road Forward Isn't Obvious

Some researchers believe the solution is fundamental architecture changes. Novel approaches to training. Systems that bake in uncertainty at a deeper level rather than patching it on top.

Others think we need smaller, more specialized models. Instead of one general system that's confident about everything, multiple focused systems that know their limitations.

The most honest answer? We don't know yet.

What we do know is that the current trajectory is unsustainable. You can't build reliable systems on top of unreliable foundations, no matter how confident those foundations sound. Every healthcare AI that recommends wrong treatments. Every legal system using AI to evaluate evidence. Every hiring tool filtering resumes. They're all built on this assumption that we can make hallucinations "good enough."

We can't. Not while we're optimizing for confidence instead of calibration.

If you want deeper context on this problem, check out How AI Learned to Gaslight: The Rise of Synthetic Confidence in Large Language Models, which explores how this issue manifests in production systems.

The uncomfortable truth is that we've built something genuinely impressive that's also increasingly unreliable. And we've built it in a way that makes it harder to notice the unreliability. That's not progress. That's just a more sophisticated way of failing.