Photo by Microsoft Copilot on Unsplash

Last week, I asked ChatGPT to write a historical summary of the Battle of Hastings. It produced three eloquent paragraphs—complete with specific dates, troop numbers, and analysis—that sounded absolutely authoritative. There was just one problem: about 40% of it was fabricated. Not in an obvious way. The model didn't say aliens invaded or that Napoleon fought at Hastings. Instead, it wove fictional details seamlessly into a narrative that felt real enough to convince someone who didn't know better.

This phenomenon happens constantly, and it reveals something unsettling about how modern AI systems actually work. They're not search engines looking up facts. They're pattern-matching machines that have learned to sound convincing, whether or not they're actually correct. And that distinction matters more than most people realize.

The Confidence Problem Nobody Warned You About

Here's what makes this tricky: an AI model doesn't "know" it's making something up. It has no internal alarm bell. When you ask a language model a question it hasn't seen before, it generates tokens (essentially word fragments) one at a time based purely on statistical probability. The model predicts what word should come next based on patterns it learned during training.

If those patterns suggest that historical summaries tend to have dates in them, the model will generate a date. It might be real. It might be invented. From the model's perspective, there's no functional difference. Both feel equally "correct" according to its training data.

A 2023 study from the University of Michigan found that ChatGPT hallucinates—generates false information—in roughly 3% of its responses across straightforward factual questions. That sounds small until you realize it compounds across multiple questions. Ask it ten questions, and statistically, you might get one completely fabricated answer delivered with perfect confidence.

The real kicker? The model tends to sound *more* confident when it's wrong. Researchers at Stanford discovered that language models often use more definitive language and fewer hedging phrases when generating hallucinations. It's as if the confidence itself makes the falsehood more convincing to itself.

Why This Matters More Than You Think

Most people understand that AI isn't perfect. But there's a difference between "occasionally wrong" and "can't distinguish between facts and fiction in its own outputs." The latter describes the current state of large language models.

Consider a lawyer using an AI to research case precedents. Or a doctor consulting an AI for diagnostic information. Or a student asking an AI to summarize a complex scientific concept for a paper. Each of these scenarios involves someone relying on information they can't immediately verify, taking the AI's confident tone as evidence of accuracy.

A particularly troubling case emerged in 2023 when a lawyer submitted a brief citing court cases that ChatGPT had invented. The citations looked real. They had case numbers, court names, and everything. The opposing counsel called them out, but only because they checked. How many cases have gone unnoticed?

Then there's the compounding effect. These hallucinations get shared, repeated, and embedded in new datasets. An AI trained on internet text that includes AI-generated hallucinations might learn to replicate those same false facts, essentially baking errors into the next generation of models.

The Tools Nobody's Really Using Yet

Some organizations are working on solutions, though adoption is still embarrassingly low. One approach involves training models to express uncertainty. Instead of generating a confident answer, the model learns to say "I'm not sure about this" or "I don't have reliable information on that." It sounds worse from a user experience perspective, but it's actually more honest.

Another method uses "retrieval augmentation." Rather than generating responses from pure pattern matching, the AI first searches a verified database or knowledge base, then generates an answer based on what it actually found. This is why some AI systems ask you what documents to search—they're trying to ground their responses in reality.

A third approach involves ensemble methods, where multiple models generate responses and the system flags disagreements. If five AI models give five different answers, that's a red flag worth investigating.

Yet most people using AI aren't using any of these approaches. They're just asking ChatGPT or Gemini or Claude directly and hoping for the best. Which, statistically speaking, usually works. But "usually" isn't good enough for some use cases.

What Actually Works Right Now

If you're going to use AI for anything important, treat it like you'd treat a smart person you've just met and don't fully trust. Here's what that means in practice:

First, verify anything factual before using it. Ask the AI for sources. If it can't provide them or the sources don't check out, don't use the information. Second, ask follow-up questions. Ask the AI the same question in a different way and see if you get the same answer. Inconsistency suggests hallucination. Third, use AI for thinking, not just answering. Use it to brainstorm or explore ideas, then verify the actual facts elsewhere.

Fourth, pay attention to confidence signals. When an AI says "I'm not certain, but..." or "based on my training data..." or "I should note that I can't verify this," that's a model being relatively honest about its limitations. Value that.

If you want a deeper understanding of where AI problems really originate, read about why AI models hallucinate and what we're learning from their mistakes.

The Uncomfortable Truth

We're in a strange period where AI has become capable enough to be useful but not reliable enough to trust completely. That's actually fine. Most tools have limitations. The problem is that AI's limitations are often invisible. A chainsaw looks dangerous. A faulty AI recommendation looks normal.

As these systems become more integrated into our work and lives, the real skill isn't learning how to use AI. It's learning how to use AI skeptically. That's harder than it sounds, especially when the AI is very, very good at sounding like it knows what it's talking about.