Photo by Steve Johnson on Unsplash

Last month, an AI system confidently told a user that Abraham Lincoln invented the airplane. The week before, another one fabricated an entire academic paper, complete with fake citations and a made-up author. These aren't glitches or edge cases anymore. They're becoming the rule.

If you've interacted with ChatGPT, Claude, or any large language model lately, you've probably encountered this phenomenon: the AI hallucinating facts with absolute certainty. It doesn't seem hesitant. It doesn't hedge its bets. It lies with conviction.

The weird part? The models are getting worse at this, not better. And the reason why reveals something uncomfortable about how we've been building these systems.

The Confidence Problem Nobody Predicted

When OpenAI first released GPT-3 in 2020, hallucinations existed, but they seemed like a fixable problem. Engineers assumed that as models got larger and trained on more data, accuracy would improve. More parameters, more training data, more compute power—this was the formula that had worked for nearly everything else in deep learning.

It didn't work.

Researchers at Stanford and Berkeley discovered something counterintuitive in 2023: larger models actually hallucinate more frequently than smaller ones, at least in certain domains. A 7-billion parameter model might get a factual question right 60% of the time. Scaling it up to 70 billion parameters? Sometimes it drops to 45%.

The issue isn't capability—it's confidence. These massive models have learned to generate fluent, contextually appropriate responses so well that they've also learned to sound absolutely certain about things they actually have no way of knowing. They've optimized for one thing: sounding right. Not being right.

During training, when a model generates plausible-sounding text that matches the pattern of correct answers, it gets rewarded. The model doesn't have an internal fact-checking mechanism. It doesn't know the difference between "I read this in a training document" and "I'm making this up because it sounds like something that could be true."

Why Your AI Sounds Like It Just Googled Everything

Here's what happens inside the black box: A language model is essentially a next-word prediction machine. It's been trained on terabytes of internet text, books, articles, and other human-generated content. When you ask it a question, it's not retrieving information like a database would. It's generating the most statistically likely sequence of tokens that would follow your question, based on everything it learned during training.

That's powerful for creative tasks and surprisingly good for explaining concepts. But for factual accuracy, it's a disaster waiting to happen.

Imagine training someone to write in the style of Wikipedia by showing them thousands of Wikipedia articles. Eventually, they'd get really good at sounding like Wikipedia. They'd know the structure, the tone, the formatting. But if you asked them a question about something that wasn't in their training data, or something that's changed since their training ended, they wouldn't say "I don't know." They'd confidently invent an answer that sounds Wikipedia-like.

That's essentially what's happening. The larger the model, the more sophisticated the mimicry, and the more convincing the hallucinations.

Companies have tried various fixes. Temperature adjustments, prompt engineering, retrieval-augmented generation (RAG) where you feed the model external data to reference. These help, but they're band-aids on a fundamental architectural problem. The core issue remains: these models are probability machines, not knowledge stores.

The Scaling Wall We're Pretending Doesn't Exist

There's a theory in AI circles that's becoming increasingly hard to ignore: we might be hitting a hard limit on how much we can improve these systems by just throwing more compute at them. The era of "bigger equals better" might actually be over.

OpenAI's recent pivot toward reasoning-focused models like o1 suggests they're thinking about this problem. Instead of making models that predict the next word faster and faster, they're experimenting with models that actually work through problems step-by-step, almost like showing their work in a math problem.

Early results are intriguing. These reasoning models hallucinate less. But they're also slower and more expensive to run. It's the opposite direction of where everyone thought we were heading.

The uncomfortable truth is that the easy wins in scaling are behind us. Getting from GPT-2 to GPT-3 to GPT-4 to GPT-4 Turbo followed a predictable curve. But the jump from "very good at sounding right" to "actually accurate" might require fundamental rethinking of architecture, training methodology, or both.

What This Means for Your AI Interactions

If you're using AI for anything remotely important—research, decision-making, factual information—you need to treat it like a brainstorming partner, not a search engine. It's phenomenally good at generating plausible text. It's terrible at guaranteeing truth.

The models know this, by the way. Ask them directly: "Are you prone to hallucinations?" They'll happily admit it. They just won't stop doing it. Because the incentive structure during training didn't select for honesty about uncertainty—it selected for fluent, confident-sounding responses.

This connects to a broader pattern in how AI systems learn biases and behavioral quirks. Understanding why your AI chatbot keeps apologizing reveals deeper issues about how these systems absorb and reflect human patterns. Hallucinations aren't separate from those problems—they're part of the same underlying challenge: AI systems optimizing for the wrong things.

The Future Might Require Honest Uncertainty

The path forward probably involves training models to be comfortable saying "I don't know." Revolutionary, right? But that's actually a dramatic departure from current approaches. During training, models that confidently generate plausible answers get rewarded. Models that frequently say "I don't know" get penalized because those responses look wrong to the optimization algorithm.

Fixing hallucinations might mean accepting slower, less impressive-sounding systems. It might mean hybrid approaches where AI handles generation and human systems handle verification. It might mean completely different architectures that we haven't invented yet.

What's certain is that we can't scale our way out of this problem. The emperor has no clothes, and the emperor is really, really confident about it.