Photo by Luke Jones on Unsplash

Last Tuesday, I asked ChatGPT to tell me about the mayor of my hometown. It generated three paragraphs about "Mayor Patricia Holloway," complete with her background in urban planning and her 2019 election victory. There was just one problem: she doesn't exist. Neither do any of the initiatives she supposedly championed.

I already knew this. I was testing something I'd suspected for months: that modern AI systems are fundamentally designed to sound right rather than to *be* right. And they're disturbingly good at it.

The Confidence Problem Baked Into the Design

Here's the uncomfortable truth that nobody in Silicon Valley really wants to say out loud: language models aren't truth-seeking machines. They're pattern-matching engines trained on billions of words, and they're optimized for one metric above all others—producing text that sounds like it belongs in the training data.

When you ask GPT-4 or Claude or any of their cousins a question, the model doesn't actually "think" about whether it knows the answer. Instead, it calculates: what sequence of words would most likely follow this prompt, based on everything I've seen during training? If you ask about a real event, great. If you ask about something obscure or niche, the model will happily generate plausible-sounding fiction because, from a statistical perspective, confidence-laden nonsense often gets similar training signals to actual knowledge.

The researchers call this "hallucination." I think that's a generous term. Hallucination implies the model is experiencing something it can't control. Really, it's just doing exactly what it was trained to do—maximize the probability of the next token in the sequence, regardless of whether that token exists in any meaningful reality.

A 2023 study from the University of Washington found that when language models don't know something, they're actually *more* likely to sound confident about it than when they do know something. The reason is simple: training data contains plenty of confident statements about real things (because people write confidently about what they know) but also tons of confident statements about fake things (because people generate plausible-sounding filler constantly). The model learns that confidence is just... part of how text is structured.

Why Hallucinations Feel More True Than Facts

The truly sinister part? AI bullshitting is often *more convincing* than the truth.

Take my fictional Mayor Holloway. The model didn't just make up a name—it fabricated a coherent narrative. It mentioned specific years, policy areas, and political context. It used the language patterns of a real biography. Your brain, when reading this, activates the same pattern-recognition systems that normally help you trust credible sources. Specificity triggers credibility. Details create coherence.

This phenomenon has been studied extensively. Research suggests that when something has internal logical consistency—when claim B follows from claim A in a way that makes sense—people trust it more, regardless of whether claims A and B are actually true. AI systems are *exceptional* at generating internally consistent fiction.

Compare this to how a human expert usually acts. When I ask my neighbor (a history professor) something outside her expertise, she'll often say "I'm not sure, but..." She'll express uncertainty. She'll hedge. She'll sound less authoritative because she's genuinely unsure. Meanwhile, the language model will deliver Mayor Holloway with complete syntactic confidence, and your brain will eat it up.

The Real Problem With AI Reliability Right Now

The bigger issue than individual hallucinations is what happens at scale. Most people use AI tools for things that *seem* like they should be safe—summarizing articles, brainstorming ideas, writing boilerplate code. But these are exactly the contexts where hallucinations go unnoticed.

A lawyer in New York famously used ChatGPT to cite case law. The model cited several cases that didn't exist, complete with fake legal citations. The lawyer submitted these citations in court filings, creating an absolute nightmare that nearly got him disbarred. When he later checked, he realized the model had confidently invented jurisprudence.

This wasn't the model malfunctioning. It was doing exactly what it does—generating plausible case citations based on the statistical patterns in its training data (which included discussions of countless real court cases). From the model's perspective, making up a case is indistinguishable from remembering one.

The problem scales when you consider how many people now delegate research to these systems. A marketing team uses Claude to write product descriptions based on specifications. An HR department uses an AI tool to summarize candidate backgrounds. A student uses GPT to understand historical events. Each time, the system is generating confident-sounding text that *might* be based on real patterns from its training, or might be pure interpolation. You won't know the difference without external verification.

What You Can Actually Do About It

Understanding this problem is the first step to not falling for it. Before treating any AI output as fact, ask yourself: is this something I can verify? If the answer is no, you shouldn't trust it—not because AI is evil or broken, but because it's not designed to be reliable about things you can't check.

When you're asking AI about subjective things ("what's a creative idea for a campaign?"), hallucinations matter less. The system is surfacing patterns from human creativity, which is largely what you want. When you're asking about objective facts that matter ("what's my legal liability in this situation?"), treating AI output as a starting point for human verification rather than a final answer is non-negotiable.

You should also read Why Your AI Chatbot Confidently Lies to You (And How to Spot When It's Making Things Up) for specific techniques on identifying when an AI is probably hallucinating in real time.

The uncomfortable reality is that we've built incredibly fluent systems that have no actual concept of truth. They're pattern-prediction machines wearing the mask of intelligence. The confidence they project? That's not a feature. It's a bug that looks exactly like a feature.

And we're all just starting to realize it.