Photo by Microsoft Copilot on Unsplash

Last Tuesday, I asked ChatGPT who won the 1987 World Series. It told me it was the Minnesota Twins. Confidently. With the kind of certainty that makes you believe it. The only problem? The Minnesota Twins won in 1987, 1991, 2002... wait, no. They actually won in 1987 and 1991. But I didn't know that immediately, and I suspect many people just accepted the answer and moved on.

This is the real crisis nobody's talking about. It's not that AI gets things wrong—it's that AI gets things wrong while sounding absolutely, unequivocally right.

The Confidence Trap

When you talk to a language model, you're interacting with something that has no concept of certainty the way humans do. A language model doesn't "know" things. It predicts the next word based on patterns in training data. But—and this is crucial—the architecture that makes these models work has no built-in mechanism for expressing doubt.

Think about how humans communicate. When we're sure about something, we say it one way. When we're uncertain, we soften our language. "I think it was around 1987." "I'm pretty sure, but don't quote me." "Actually, I have no idea." These verbal hedges are native to human speech because we genuinely experience varying degrees of confidence.

Language models? They just keep outputting tokens. They don't have an internal experience of confidence. They have something that looks like confidence to you because they were trained on human text that contains confident statements. So they've learned to produce confident-sounding text. But there's no "belief" underneath it—just statistical probability grinding away.

According to research from OpenAI, GPT-4 has a 92% confidence rating when it answers questions about topics it's actually trained on—yet it still makes errors about 8% of the time. That doesn't sound so bad until you realize that humans don't confidently assert things they're uncertain about 8% of the time. We'd say we weren't sure.

Why This Matters More Than You Think

The consequences of this mismatch are becoming impossible to ignore. A lawyer in New York used ChatGPT to research case law for a brief. The AI cited six cases. Five of them didn't exist. Not "difficult to find"—genuinely fabricated, with proper citations that looked entirely legitimate. The lawyer got sanctioned by the court. ChatGPT had done exactly what it was designed to do: generate plausible-looking text. It just happened to be entirely fraudulent.

Medical professionals have reported similar incidents. Someone asks an AI about symptoms, gets a confident diagnosis that sounds official, and makes healthcare decisions based on an invention. Students use AI to help with research and end up citing studies that were never published. Journalists report statistics from confident-sounding AI that have no basis in reality.

Here's the thing that keeps me up: these mistakes aren't the ones that get caught. Somebody fact-checks the lawyer's citations and the fraud comes to light. But for every one of those cases, there are thousands—probably millions—where people accept the confident-sounding answer and never verify. A startup founder gets business advice from an AI. An engineer makes design decisions based on AI suggestions. A student learns history from an AI tutor. None of them cross-check every claim because the AI said it with such certainty.

For more on this phenomenon, you might want to read about why AI chatbots confidently argue with you about facts they just made up—it goes deeper into the mechanics of how models construct convincing falsehoods.

The Technical Reason This Is So Hard to Fix

You might think the solution is simple: just make AI less confident. Make it hedge its bets. Add uncertainty tokens.

Developers have tried. It doesn't work the way you'd hope. When you explicitly train models to express uncertainty, they learn to do it about everything. A model trained on "I'm not certain, but..." starts prefixing all its responses with disclaimers, even when discussing basic facts. The user experience becomes painful. People stop using the tool.

There's also a fundamental problem at the architecture level. These models are trained through something called "next token prediction." Given a sequence of words, predict the next word. Then the next one. Then the next. It's remarkably good at this task, which is why these models work at all. But the task itself has no mechanism for saying "stop, I don't know this part." It just keeps predicting. The model would need a fundamentally different training process to develop genuine uncertainty—one where it could learn to say "I don't know" and be rewarded for that honesty.

Some teams are working on this. OpenAI's research into "interpretability" aims to understand what's happening inside these models. Anthropic is developing Constitutional AI methods that try to align model behavior with human values, including honesty about limitations. But these are early-stage efforts, and deploying them at scale takes time.

What You Should Actually Do

Until—if ever—we solve this problem at the architecture level, the responsibility falls on users. And I don't say that to blame you. I say it because it's the truth we have to live with right now.

Treat AI output like you'd treat a Wikipedia article written by someone you don't know. It's a starting point, not a conclusion. Verify claims, especially important ones. If you're making decisions that matter—medical, legal, financial, educational—don't base them on what an AI said without checking. Check multiple sources. Look for citations and verify them.

And maybe most importantly: the more confident something sounds, the more carefully you should scrutinize it. That's not how human speech works, but it's absolutely how language models work.

The AI revolution is real and these tools are genuinely useful. But we're living through a transition period where the capabilities and the reliability haven't caught up with each other. The confidence gap is real. And until it narrows, that gap between how sure these systems sound and how sure they actually are? That's where the risk lives.