Photo by Igor Omilaev on Unsplash

Last month, a lawyer in New York filed a legal brief citing six cases that never existed. The citations looked perfect. The case names were plausible. The legal precedents seemed solid. All of them were invented by ChatGPT.

This wasn't an isolated incident. A doctor asked an AI chatbot about a rare disease and received a detailed explanation of treatment options. When he checked the medical literature, half the recommendations came from studies that don't exist. An engineer used an AI tool to debug code and ended up spending six hours implementing a "solution" that the AI created from thin air.

We call these hallucinations. But that word might be doing us a disservice.

The Hallucination Is Actually the Feature

When you ask a large language model a question, it doesn't retrieve facts from a database. It doesn't check Wikipedia. Instead, it performs statistical pattern matching across billions of text examples. It's predicting the next token—the next word or piece of a word—based on probability.

The system has learned patterns so well that it can generate text that looks and sounds like it should be true. It strings together words in ways that feel authoritative and coherent. This is the same mechanism that makes these models useful in the first place. The ability to generate human-like text, to continue a conversation naturally, to write code that works—all of these flow from the same underlying process.

So when the model generates false information with absolute confidence, it's not breaking. It's working exactly as designed. The problem is that confidence and accuracy aren't the same thing.

Think about how a human conversation works. If I tell you "Napoleon was defeated at Waterloo in 1815," you believe me because the statement sounds authoritative and fits with what you already know. Now imagine if every single statement I made, whether true or false, came out with that same tone of certainty. You'd have no way to distinguish fact from fiction without checking every claim.

That's the experience of using an AI chatbot. Why Your AI Chatbot Becomes Dumber When You Ask It the Right Questions explores this phenomenon further, showing how more specific or challenging queries can paradoxically trigger worse outputs.

Where the Confidence Comes From

Here's what makes this particularly insidious: the model's confidence isn't random. It's proportional to how well the pattern matches something in the training data. If you ask about World War II, the model has seen thousands of documents about it and generates relatively reliable information. Ask about a niche scientific topic, and the model might still sound confident, but it's working with far fewer examples to draw from.

The researchers at OpenAI have measured this. They found that as you ask more complex or specialized questions, the probability of hallucinations increases—but the model's confidence levels stay roughly the same. The system has learned to sound sure about everything, regardless of whether it actually knows the answer.

This creates a particularly nasty failure mode. A doctor or lawyer using these tools might trust them more in edge cases precisely because they're too busy to verify every detail. These are exactly the situations where hallucinations are most likely to occur.

The Economic Incentive to Make Worse Models

There's something perverse happening in the AI industry right now. The models that are most useful are the ones that hallucinate most confidently. They generate longer responses. They never say "I don't know." They give you an answer when you need one, even if that answer is completely made up.

Users love this. It feels good to get a complete response. It feels authoritative. Companies love it because engagement metrics go up. Someone asking ChatGPT to write an entire article gets a complete article. Someone asking for help debugging gets a full code solution. These interactions feel more useful than a model that constantly hedges and admits uncertainty.

But what if the best user experience is actually worse for society? What if we've optimized for confidence instead of correctness?

The fix should be simple: make models that say "I don't know" more often. Add uncertainty quantification. Make the system admit when it's extrapolating. But these features make the user experience feel worse. The responses are shorter. The interactions feel incomplete. In some early experiments, users actually rated these more honest models as less helpful.

What's Actually Changing

The latest generation of models is attempting something different. GPT-4 and similar systems are being trained with human feedback that prioritizes accuracy over engagement. They're being given external tools to look things up. They're being fine-tuned to refuse questions they can't answer reliably.

It's working, sort of. Hallucination rates are going down. But the improvement is slower than you'd expect, and it comes at the cost of reduced capability in areas where the model was actually useful. You can't have both a system that's endlessly creative and a system that's reliably accurate. At some point, you have to choose.

The real solution probably isn't in the models themselves. It's in how we deploy them. AI should be used as a research assistant, not an oracle. It should be a tool that suggests possibilities, not a source of ground truth. The moment someone treats an AI output as verified fact without checking, the system becomes dangerous—not because it's breaking, but because it's working exactly as intended.

The Uncomfortable Truth

What bothers many AI researchers isn't that these systems hallucinate. It's that we don't actually have a great explanation for why, or a clear path to stopping it without making them fundamentally less capable. The systems are operating at the boundary of what's possible with current technology. The confidence without knowledge isn't a bug we can patch. It's inherent to how statistical language models work.

Until we're honest about that limitation, we'll keep deploying these tools in situations where their weaknesses matter most. And we'll keep being surprised when a lawyer ends up in court, or a doctor makes a diagnosis, or an engineer ships broken code—all because an AI sounded absolutely sure of itself.