Photo by Nahrizul Kadri on Unsplash

Last month, I asked ChatGPT who won the 2019 Super Bowl. It told me the Kansas City Chiefs defeated the San Francisco 49ers 31-20. That's correct. Then I asked it again with a slightly different phrasing. This time it said the New England Patriots won 37-24. Both answers came with the same unwavering confidence, delivered in the same authoritative tone. One was right. One was completely fabricated. The AI had no idea which was which, and frankly, it didn't care.

This is the uncomfortable truth nobody wants to talk about at tech conferences: we've built artificial intelligence systems that are fundamentally unreliable, yet we've trained them—and ourselves—to sound absolutely certain about everything.

The Confidence Trap

Large language models work by predicting the next most statistically probable word based on everything that came before it. They're not reasoning. They're not checking facts. They're not second-guessing themselves. They're performing an elaborate statistical parlor trick that happens to produce grammatically correct sentences that sound like they come from someone who actually knows what they're talking about.

A 2023 study from Google and DeepMind found that state-of-the-art language models hallucinate—that is, confidently state false information—roughly 3-5% of the time in factual domains. That might sound small until you realize you're interacting with a system making thousands of micro-decisions per response. The errors compound. They snowball. And they always sound plausible.

Here's what makes this worse: we humans are terrible at detecting when AI is lying. Research from Stanford showed that people trust AI summaries and answers at significantly higher rates than human-generated content, even when the AI output is demonstrably wrong. There's something about the crisp formatting, the grammatical perfection, the lack of hedging language that makes our brains go "yeah, that tracks." We outsource our skepticism to punctuation.

The really insidious part? These systems don't just fail randomly. They fail in predictable patterns. They tend to confidently assert information about recent events (which fall outside their training data). They struggle with numerical reasoning but compensate by producing plausible-sounding numbers. They confidently make up citations that sound like they could exist. They're not broken in charming, obviously-wrong ways. They're broken in sophisticated ways that require actual expertise to detect.

Why Your Chatbot Sounds Like a Know-It-All

The architecture of modern language models almost guarantees overconfidence. When you ask them a question, they're not accessing a database of facts. They're generating text based on statistical patterns learned from the internet—a place where confident, wrong people vastly outnumber humble, correct ones. The training data is flooded with bullshit articulated beautifully.

Additionally, these systems lack what researchers call "epistemic humility." They can't say "I don't know" in any meaningful way. They can be programmed to output the words "I don't know," but they have no internal model of their own uncertainty. They have no gut feeling, no sense of being out of their depth, no alarm bells. When they hit the probabilistic equivalent of thin ice, they just keep walking.

We've also trained them to sound authoritative because, frankly, that's what we rewarded them for. OpenAI's RLHF (Reinforcement Learning from Human Feedback) process used human raters to improve model outputs. Those raters consistently preferred confident, detailed answers over cautious, hedged ones—even when the hedging was warranted. We selected for fluency and polish, not for epistemic accuracy.

The result is that confidence became orthogonal to correctness. A model learned that writing with certainty gets better ratings, regardless of whether the content is true. You can read more about this phenomenon in our article on how AI learned to sound confident while being completely wrong.

The Professional World Is Getting Fooled

This wouldn't be merely a parlor trick concern if people weren't making real decisions based on AI outputs. But they are.

A lawyer in New York famously submitted a brief to court citing cases generated entirely by ChatGPT—cases that didn't exist. The judge was not amused. A researcher published findings based on data analysis performed by GPT-4, only to discover later that half the statistical claims were invented. A marketing team took medical claims generated by an AI and published them without verification; they got sued.

The problem scales with the stakes. In domains where you have domain expertise, you can usually catch the BS. Ask a language model about your field of expertise and you'll immediately spot the hallucinations. But ask it about something you don't know well? You're flying blind, trusting a system that has no capacity for self-awareness about what it doesn't know.

What Actually Needs to Change

This isn't an argument for banning AI or declaring it useless. It's an argument for radical honesty about what these systems are and aren't.

First, we need to stop training models to be confident. We should be selecting for models that explicitly flag uncertainty, that hedge appropriately, that say "I'm not sure, but here are some possibilities" instead of presenting speculation as fact. Yes, that makes for less satisfying interactions. That's the point.

Second, we need better retrieval-augmented generation systems—AI that doesn't just generate text from training data, but actually cites and verifies sources in real-time. Some companies are building this, but it's not the default.

Third, we need regulatory requirements that prevent high-stakes decisions from being made based solely on AI outputs without human verification. Your insurance claim denial probably shouldn't be decided by a model that hallucinates 5% of the time.

Most critically, we need cultural shift. Every time someone uses AI to write something important—a report, an email to a customer, a policy document—they should treat it like a first draft from an intern: smart, capable, but needs fact-checking. We've been treating it like wisdom from an oracle. That's the real problem.

The technology itself isn't malicious. The systems don't know they're lying. But that's almost worse. At least malicious liars can be held accountable. Confident hallucinations are just passing through our society, shaping decisions and spreading misinformation, wearing the mask of certainty.