Photo by Google DeepMind on Unsplash

Last Tuesday, I asked ChatGPT who won the 2019 World Series. It told me it was the Houston Astros. Confident. Detailed. Completely wrong. The answer was the Washington Nationals, and the AI didn't just miss—it doubled down when I challenged it, explaining the Astros' "dominant pitching rotation" that season. That's the thing about modern AI systems: they don't just make mistakes. They make mistakes with an almost unsettling certainty.

This isn't a bug in the system. It's the foundation of how these models actually work. Understanding why requires us to abandon the idea that AI assistants are "thinking" in any meaningful sense. They're not. They're doing something much stranger.

The Confidence Illusion: What's Really Happening

When you interact with a large language model, you're watching a system that has learned to predict the next word in a sequence, billions of times over. During training, these models consumed vast amounts of text from the internet—accurate Wikipedia articles sitting right next to conspiracy theories, published research alongside confident nonsense from random blogs.

The model didn't learn the difference between true and false. It learned patterns. It learned which word combinations appear frequently together. If your training data includes 10,000 articles saying "the Earth orbits the sun," but also includes fringe websites claiming "NASA hides the truth," the model learns both patterns equally well. It becomes exceptionally skilled at reproducing language that *sounds* authoritative, regardless of whether the underlying claim is accurate.

Here's where it gets disturbing: a model trained on text that's 99% accurate will still hallucinate with supreme confidence. A Stanford study found that GPT-3 confidently produced false information about 3-5% of the time on factual questions. That might sound low until you realize what it means in practice. If you ask it 20 questions about specific facts, expect it to lie to you on one of them—with no hesitation, no uncertainty markers, just pure synthetic confidence.

The Illusion of Understanding

The real problem isn't that AI models lack information. It's that they lack something far more fundamental: they don't actually understand what truth *is*. They've never had a genuine belief about anything. They've never been wrong and felt the sting of correction. They've never changed their mind because evidence compelled them to.

Consider what happens when you ask an AI system a question outside its training data—something genuinely novel. A human in the same position would typically express uncertainty. "I'm not sure," they might say. An AI? It generates an answer anyway, because that's what it was optimized to do. It was trained to maximize the likelihood of producing plausible-sounding text, not to know when to shut up.

This is exactly why AI chatbots sound confidently wrong: they're optimized for fluency, not accuracy. The systems that sound the most authoritative are often the ones most likely to be fabricating details you'll never catch.

Why More Training Data Made Things Worse

You'd think scaling these models up—feeding them more data and more computing power—would make them more accurate. And in some narrow domains, it has. But there's a creeping problem that researchers are only now fully grappling with: more data means more misinformation to learn from.

OpenAI's own research shows that when they scaled up GPT-2 to GPT-3 to GPT-4, the models became better at many tasks, but they didn't become more honest. They became better at *sounding* honest. They learned more sophisticated ways to package falsehoods in authoritative language. A 7 billion parameter model might have hedged its incorrect claims with phrases like "it's possible that..." A 175 billion parameter model? It states its fabrications as fact.

The model is essentially learning to imitate the confident bullshitter rather than the humble expert. And the internet has no shortage of confident bullshitters to learn from.

What This Means for You Right Now

If you're using AI assistants for anything that matters—research, decision-making, professional work—you need to treat them like you'd treat a well-spoken stranger at a party. They might sound knowledgeable. They probably aren't lying intentionally. But they also have no accountability for what they're saying, no internal experience of doubt, and no genuine concern for accuracy.

The most dangerous outputs from these systems aren't the ones that are obviously wrong. They're the ones that are partially true, mixed with false details that sound plausible. If an AI tells you something that matches what you already believe, your brain will accept it without question. That's not the AI's fault. That's how human cognition works. But it's worth knowing you're vulnerable to that particular attack.

Some organizations are building "confidence scoring" systems that attempt to measure how certain a model actually is about a claim. Others are creating retrieval-augmented generation systems that force the model to cite specific sources and check them. These are steps in the right direction. But they're band-aids on a fundamental problem: we asked computers to predict the next word really well, and we're shocked that they're not great at understanding the truth.

The Future Probably Isn't What You Think

There's a persistent belief that if we just train bigger models, use better data, and refine the techniques, we'll eventually get AI assistants that are both capable *and* truthful. Maybe. But the current architecture of these systems has inherent limitations that might not be solvable through scale alone. You can't get a system to care about truth if it was never designed to understand what truth means in the first place.

The most valuable versions of AI assistants in the near future probably won't be the ones that try to answer everything with confidence. They'll be the ones that know when to say "I don't know," that cite sources you can verify, and that admit uncertainty when the evidence is mixed. They'll be less impressive-sounding, less polished, less like talking to an expert. And they'll be far more honest.

In the meantime? Verify anything important. Check the sources. Ask follow-up questions. Treat AI as a research assistant that sometimes lies, not as an oracle. It's more work than trusting it completely. But the alternative—confidently acting on information that might be completely fabricated—is significantly worse.