Last Tuesday, I asked ChatGPT who won the 2019 World Series. It confidently told me it was the Boston Red Sox. They didn't. The Houston Astros did. When I corrected it, the chatbot apologized and "thanked me for the feedback"—as if it would remember this interaction next time, which it won't.

This moment crystallizes something that frustrates millions of people: why do these supposedly intelligent systems say false things with such conviction? The answer isn't that AI companies are cutting corners or that the technology is fundamentally broken. It's far more interesting than that.

The Prediction Machine Masquerading as Knowledge

Here's what ChatGPT and similar large language models actually do: they predict the next word in a sequence. That's it. They're extraordinarily sophisticated at this task—trained on hundreds of billions of words from books, articles, and websites—but prediction isn't the same as understanding.

Think of it like this: if you read enough mystery novels, you'd get very good at predicting plot twists. You could generate plot points that sound plausible and internally consistent. But predicting what comes next in a pattern isn't the same as solving a murder. The model has never actually investigated anything. It has never verified information against reality.

When an AI system generates false information, it's not lying in the way humans lie. Lying requires intent to deceive and knowledge that you're being deceptive. The chatbot doesn't know it's wrong. It simply calculated that certain words were statistically likely to follow your question, based on patterns in its training data. If incorrect information appeared frequently in that training data (like myths, outdated information, or just statistical noise), the model will replicate it.

Researchers at Anthropic found that as they scaled up language models to be more powerful, they became better at sounding confident—even when they were completely wrong. A smaller model might hedge its bets with "I'm not sure, but..." A larger model confidently states falsehoods because the statistical patterns it learned reward certainty over hedging.

When Training Data Becomes Training Quicksand

The specific type of wrongness these systems produce tells us a lot about their nature. They're particularly prone to what researchers call "hallucinations"—generating facts that sound plausible but are entirely fabricated.

I tested this myself. I asked Claude to give me citations to papers on a specific narrow topic in neuroscience. It provided five references with authors, years, and journal names that sounded completely legitimate. I checked them. None existed. Not a single one. But they were formatted correctly. They had realistic author names and plausible titles. The model had essentially been asked to generate academic citations, and it did—by predicting what academic citations look like, not by accessing an actual database of papers.

This happens because the training data is finite. If something wasn't in the training data or was underrepresented in it, the model has learned a probabilistic sketch of what it "should" look like based on patterns. When asked about it, the model fills in the gaps by predicting reasonable-sounding details. The problem is, reasonable-sounding isn't the same as true.

Training data recency is another culprit. Most large language models have knowledge cutoff dates. GPT-4's training data includes information through April 2023. Ask it something that happened in August 2024 and it will either refuse to answer or, worse, confidently fabricate something. The model doesn't know what it doesn't know—it just generates plausible-sounding text.

The Confidence Problem We Haven't Solved

The most insidious issue isn't inaccuracy—it's unwarranted certainty. Humans are often wrong too, but we usually have metacognitive awareness of what we don't know. I know the difference between something I'm confident about and something I'm guessing at. I know I'm speculating.

Language models don't have this internal quality control. They can't distinguish between common knowledge and rare knowledge, between verified facts and speculation that happens to be statistically coherent. They output with the same grammatical confidence whether they're describing basic physics or inventing entirely fictional academic papers.

Engineers have tried various solutions. Adding confidence scores doesn't work well—the model often just learns to assign high confidence to things it should be uncertain about. Building in "I don't know" responses helps but degrades the system's ability to synthesize information creatively, which is one of its actual strengths.

Some companies have started fine-tuning models with human feedback to make them more cautious. Others are experimenting with retrieval-augmented generation, where the model can search for information before answering. These help, but they're band-aids on a fundamental architectural issue.

What This Means for the Future

The crucial thing to understand is that this isn't a problem that will disappear as models get bigger or better trained. It's baked into the approach itself. You cannot get true knowledge about the world from a system that's purely learning statistical patterns in text, no matter how much text you feed it.

This doesn't mean language models are useless—they're genuinely powerful tools for brainstorming, summarizing, explaining concepts, and spotting patterns in text. But they're tools you need to treat with healthy skepticism, especially for anything that matters.

Until AI systems can actually verify their outputs against reality—until they can do something besides predict what words should come next—they'll keep confidently telling us things that sound true but aren't. The next leap forward won't come from scaling these systems bigger. It will come from fundamentally rethinking how we teach AI systems to relate to truth.