Photo by julien Tromeur on Unsplash

The Moment I Realized My AI Was Bullshitting Me

Last Tuesday, I asked Claude to write me a poem about a specific historical figure. The poem was beautiful, intricate, and completely fabricated. It referenced events that never happened, dates that were wildly inaccurate, and quoted something the person supposedly said. The worst part? It sounded absolutely authoritative. There was no hesitation, no "I'm not sure about this." Just confident, eloquent fiction.

This is what researchers call a "hallucination," though I think that word is too generous. The AI wasn't confused or having a psychotic episode. It was doing exactly what it was trained to do: generating the next statistically likely word in a sequence, regardless of whether that sequence corresponds to reality.

The problem is that this fundamental architecture—the thing that makes large language models work—doesn't care about truth. It cares about probability. And sometimes, a false statement is just more probable than the honest answer "I don't know."

How Confidence Becomes a Liability

Here's where it gets genuinely unsettling. When an AI model encounters something it hasn't seen before or doesn't have strong training data about, it doesn't say "I don't know." Instead, it follows its training: predict the next word. Then the next. Then the next. It's playing a game of linguistic continuation that can feel intelligent, comprehensive, and utterly wrong all at once.

Google's researchers found that their Gemini model would confidently provide false information about specific people, events, and products. One test found it making up entirely fictional movies and attributing them to real directors. Another revealed it providing medical advice for conditions it had clearly misunderstood. The confidence level remained constant whether the information was accurate or fantasy.

The mechanics are straightforward: language models don't actually "know" anything. They've learned statistical patterns from billions of text examples. When asked a question, they're essentially performing advanced pattern matching—finding what usually comes after similar questions in their training data. If the pattern has never occurred in their training data, they'll still try to generate something coherent. The result is an answer that looks right, sounds right, but might be completely fabricated.

Think of it like asking someone to draw a car if they've never actually seen one, only read detailed descriptions. They might produce something that vaguely resembles a vehicle, but it could have wheels in impossible places or an engine in the trunk. They wouldn't know the difference—they were just following the template of "how people describe cars."

The Real-World Consequences Are Already Here

This isn't a theoretical problem. Lawyers have already been fined and sanctioned for submitting AI-generated briefs full of fabricated case citations. A healthcare company deployed an AI that confidently recommended incorrect dosing information. Students have turned in essays packed with citations to papers that don't exist. And check When Your AI Assistant Becomes a Confident Liar: The Surprising Psychology Behind Machine Hallucinations for more on how this happens at scale.

What makes these failures particularly dangerous is the veneer of credibility. A hallucinating AI doesn't sound uncertain or apologetic. It presents false information with the same tone and structure as truth. Your brain treats it the same way—confident language activates the same trust mechanisms whether the content is accurate or invented.

One doctor reported that ChatGPT recommended a medication interaction that would actually be dangerous, but framed it so medically that they almost didn't catch the error. The model had seen enough medical texts to sound authoritative, but not enough actual medical knowledge to be safe.

Why Fixing This Is Harder Than It Sounds

You might think the solution is simple: just train AI models on only true information. But this runs into immediate problems. First, there's no perfect source of truth. Even Wikipedia has errors. Medical journals occasionally publish studies with methodological flaws. Historical interpretations change. What's "true" is often contextual.

Second, the hallucination problem is baked into how these models work. A language model that's trained to predict the next word will always be tempted to generate a plausible continuation, even when it shouldn't. Some researchers are exploring approaches like teaching models to say "I don't know" more often, or grounding them with real-time information lookups. OpenAI's newer models can now search the web before answering certain questions. But this adds latency, complexity, and cost.

There's also an uncomfortable truth: some of these hallucinations are actually profitable for companies. An AI that confidently answers every question is more useful and engaging than one that frequently says "I'm unsure." Users prefer the confident version, even if it's sometimes wrong. The financial incentives don't align with accuracy.

How to Actually Use AI Responsibly Right Now

So what do you do while we're waiting for better solutions? First, treat every AI output like you'd treat a Wikipedia article written by someone you don't know: useful as a starting point, but never as a final source. Verify everything that matters.

Second, understand that different AI systems have different hallucination rates. Models trained on curated data hallucinate less than those trained on the entire internet. Smaller, specialized models are often more reliable than massive generalist ones in their domain.

Third, ask follow-up questions. If an AI provides a specific claim—a statistic, a historical date, a scientific finding—ask it to cite its sources. Force it to point to evidence. This won't catch all hallucinations (the AI might cite sources that don't exist), but it filters out some of the worst cases.

Finally, be skeptical of confidence. When an AI says something with absolute certainty, that's actually a red flag. Reality is messier than that. The models that are honest about uncertainty are the ones being responsible.

We're living through a strange moment in technology where the most impressive-sounding systems are also the most dangerous when used carelessly. That doesn't mean we shouldn't use them. It means we need to use them smarter than we currently do.