Photo by Igor Omilaev on Unsplash

Last month, a Reddit user asked ChatGPT about persistent chest pain. The AI suggested it was likely anxiety and recommended meditation. The user ignored the advice. Three days later, he was in cardiac surgery for a blocked artery. The chatbot had no medical credentials, no access to imaging, no ability to perform an actual examination—yet it had responded with such certainty that it overrode the user's instinct to seek real help.

This wasn't a rare glitch. It was a feature of how these systems work.

The problem isn't that AI is stupid. It's that AI is too good at sounding smart while being fundamentally blind to what it doesn't know. And nowhere is this more dangerous than when people ask AI systems to play doctor.

The Confidence Problem Nobody Expected

When researchers at Stanford tested GPT-4's medical knowledge, the results looked promising at first. The model scored in the 86th percentile on the USMLE (the exam doctors take to get licensed). Headlines declared a breakthrough. AI was ready for healthcare.

But then researchers dug deeper. They asked the AI system to explain its reasoning. They tested it on edge cases. They looked at what happened when the AI didn't actually know the answer.

The findings were unsettling. The AI would generate plausible-sounding explanations for conditions it had no basis for diagnosing. It would cite medical studies that didn't exist. It would confidently state contraindications with complete certainty, even when the correct answer was "we're not sure."

This phenomenon—where AI generates false information with absolute conviction—has a name: hallucination. But that term undersells the problem. A hallucination is something you know didn't really happen. What AI does is worse. It produces confident falsehoods that feel like facts.

Dr. Timnit Gebru, an AI researcher who's studied these problems extensively, points out that this isn't accidental. Large language models are trained to predict the next word in a sequence. They're incredibly good at recognizing patterns in medical text. But pattern recognition isn't diagnosis. A model that has read thousands of case studies about cardiac problems can predict that certain symptoms tend to cluster together. That's pattern matching. It's not understanding.

Why Medical AI Is Fundamentally Different From Autocomplete

Think about autocomplete on your phone. When it guesses the next word you're typing, accuracy matters—but so what if it's occasionally wrong? You catch it and move on. The stakes are low.

Medical AI operates in a completely different arena. A wrong suggestion isn't inconvenient. It can kill someone.

Yet the technology used to build medical AI chatbots is essentially the same autocomplete engine that guesses what you're about to text. The difference is supposed to be in how we deploy it. Medical AI should come with disclaimers. It should refuse certain requests. It should be transparent about uncertainty.

Most AI systems do none of this consistently. ChatGPT's initial guidance explicitly stated it wasn't a medical provider. Yet people naturally interact with it as if it were one. Ask a question, get an answer. The format itself suggests authority, even when the disclaimer is in the fine print.

Worse, users tend to ask the most dangerous questions to AI: the ones they're worried about. Because AI feels judgment-free. It responds immediately. It doesn't make you feel silly for asking. So people with undiagnosed conditions often turn to chatbots before turning to doctors.

The Real Numbers Nobody Wants to Discuss

We don't have comprehensive data on how many people have been harmed by medical advice from AI systems. There's no national registry. The incidents aren't being systematically tracked. But the anecdotes are mounting.

In 2024, emergency rooms started reporting cases where patients arrived with preventable complications because they'd delayed seeking care after consulting AI. One case involved a teenager who had a ruptured appendix—a condition that's straightforward to diagnose with a physical exam, but one that the AI had suggested might be lactose intolerance.

Meanwhile, hospitals are simultaneously trying to integrate AI into diagnostic workflows, hoping it will help doctors catch problems faster. That goal is reasonable. AI systems can process imaging data at scale. They can flag abnormalities that humans might miss. But this requires a completely different approach than a chatbot—one that operates with human doctors in the loop, where a specialist ultimately makes the call.

The challenge is that the same companies building chatbots are also trying to build diagnostic tools. And they're using similar underlying technology. It's like expecting the same engine to power both a delivery drone and a military helicopter—different use cases entirely.

What Actually Works (And Why It's Not Popular)

There are AI systems designed specifically for medical use that work better. They're trained on specific diseases, using curated datasets, and they're built to flag uncertainty explicitly. The problem is they're not nearly as flashy or accessible as ChatGPT.

IBM's Watson for Oncology was trained to help oncologists select cancer treatments. It showed promise in research settings. But in real clinical practice, it offered treatment recommendations that clinicians didn't agree with, often because the AI was weighting data differently than human experts would.

Even specialized medical AI systems run into this same wall: they're pattern-matching machines. They're not reasoning. They're not understanding. And they can't adapt to the particular context of an individual patient's life, values, and circumstances.

The honest truth is that diagnosis requires something AI doesn't have: the ability to sit with uncertainty. A good doctor doesn't always know what's wrong. Sometimes they say "I think it's X, but we need to run tests." They hold multiple hypotheses loosely. They ask clarifying questions. They notice when something doesn't fit the pattern.

If you want more on this topic, check out our article on why AI chatbots have become overconfident—it explores the broader trust crisis in language models.

The Future of AI in Medicine (If We're Honest About It)

The path forward isn't to ban AI from healthcare. The path forward is to stop pretending AI can do things it can't.

AI is genuinely useful for processing large amounts of data. It can help radiologists review scans. It can help researchers identify patterns in medical records. It can make healthcare more efficient. But it cannot replace clinical judgment. It cannot ask the right follow-up questions. It cannot examine a patient.

The responsibility falls partly on companies to build AI systems responsibly. It falls partly on regulators to create frameworks that prevent the most dangerous uses. But it also falls on users to remember something fundamental: a chatbot is not a doctor. A confident answer isn't the same as a correct one. And your health is too important to outsource to an algorithm that's trained to sound sure of itself.

Until we're honest about these limitations, AI in healthcare will continue to be a tool that makes people feel informed while leaving them dangerously vulnerable.