Photo by Steve A Johnson on Unsplash

Last Tuesday, an emergency room doctor in Philadelphia asked an AI chatbot about treatment options for a rare drug interaction. The system responded with three detailed recommendations, cited three medical journals, and sounded absolutely certain. The doctor nearly followed the advice before fact-checking. All three recommendations were fabricated. The journals didn't exist.

This wasn't a malfunction. It was the system working exactly as designed.

The uncomfortable truth about modern artificial intelligence is that these systems are phenomenally good at one thing: sounding confident. Whether they're discussing medieval history or quantum physics, whether they actually know the answer or are completely guessing, the output reads with the same unwavering authority. Researchers call this phenomenon "hallucination." I call it dangerous.

The Confidence Problem Nobody's Talking About

When you ask most large language models a question they can't answer, they don't hesitate or equivocate. They don't say "I'm not sure" or "I should clarify that I don't have reliable information on this." Instead, they generate plausible-sounding text that feels authoritative. The system fills knowledge gaps with statistically probable-seeming information, and because the architecture that generates these responses evolved to produce grammatically coherent text, the lies sound just as fluent as the truths.

Consider a study from Stanford University published in 2023. Researchers tested GPT-4 on thousands of questions where the correct answer changed based on recent events or specific contexts the model wasn't trained on. The results were sobering: the model didn't just fail to answer these questions. It answered them with extreme confidence, explaining fictional events with the same textual fluency it used for real ones.

The statistics are particularly telling. According to analysis by AI safety researchers at UC Berkeley, large language models express high confidence in their answers roughly 75% of the time. When they're actually correct? About 85% of the time. But here's the kicker: even when they're completely wrong, they express high confidence around 70% of the time. The correlation between how confident the system sounds and whether it's actually right is remarkably weak.

This creates a trust vacuum. We've trained ourselves to interpret confidence as credibility. When a system sounds certain, we assume it knows what it's talking about. But these models have no actual understanding of what they know and what they don't. They're performing language, not reasoning.

Why This Happened (And Why It's Hard to Fix)

The root cause traces back to how we train these systems. Large language models learn by predicting the next word in a sequence based on patterns in their training data. If your training data contains confident assertions about subjects where those assertions were correct, the model learns that confidence and accuracy go together. It learns to pattern-match on authoritative tone.

The problem emerges when you combine this training approach with the vast scale of internet data. The internet contains everything: peer-reviewed research, absolute nonsense, and a lot of stuff in between. The model can't distinguish between them during training. It just learns statistical patterns. When asked about rare diseases or obscure historical events or niche technical topics, the model falls back on generating plausible-sounding text using patterns it learned from wherever those topics appeared online.

Making this better isn't straightforward. Some researchers have experimented with training models to express uncertainty. You can show a system thousands of examples where humans appropriately said "I'm not sure" and reward the model when it expresses similar uncertainty. But here's what happened: the model learned to sound uncertain in certain syntactic patterns while remaining just as confidently wrong elsewhere. It learned to express uncertainty as a stylistic choice, not as a genuine reflection of its epistemic limits.

Others have tried retrieval-augmented generation, where systems search for source material before answering. This helps, but it's slower and still doesn't solve the fundamental problem. The model can still confidently misinterpret or misrepresent the sources it finds.

The Real-World Consequences Getting Worse

As these systems become embedded in more critical applications, the stakes rise. A lawyer recently submitted court documents citing case law that the AI chatbot had hallucinated entirely. The court wasn't amused. A scientist in Malaysia used an AI system to help design an experiment, and the system confidently provided incorrect dosage calculations.

The worst part? These failures often go undetected. Most people don't fact-check everything an AI tells them. Why would they? We're building these systems into search engines, into medical decision support systems, into educational platforms. We're asking billions of people to trust systems that are fundamentally untrustworthy about their own trustworthiness.

There's a fascinating parallel with how AI learned to disagree with itself and why that's making it smarter. When you get multiple AI systems to check each other's work, they catch more errors. But that approach requires running multiple models, which is expensive and slow—not practical for most applications.

What Actually Needs to Happen

The solution probably involves fundamental changes to how we design and deploy these systems. First, we need to build systems that output confidence scores alongside answers, with honest calibration. A system should say "80% confidence" only when it's actually right about 80% of the time, not 100% of the time.

Second, we need better default behaviors for uncertainty. Instead of generating confident-sounding fabrications, models should route uncertain queries to human experts or explicitly state the limits of their knowledge. This feels like it would reduce the systems' perceived usefulness, but it's actually the only way to build real utility. A tool that sometimes fails but tells you when it's failing is more useful than a tool that fails silently.

Third, and this matters most: we need to stop deploying these systems in critical contexts without robust human oversight until we've solved this problem. Having an AI offer writing suggestions? Fine. Having it help diagnose medical conditions or influence legal decisions? Not fine. Not yet.

The uncomfortable reality is that we've created powerful text-generation systems and called them artificial intelligence. They're remarkable at some things. But they're also fundamentally limited in ways that matter deeply. The sooner we acknowledge those limits and design our systems around them, the sooner we can actually build AI we can trust.

The emergency room doctor got lucky. Others might not.