Photo by Luke Jones on Unsplash

Last month, a marketing manager at a Fortune 500 company almost submitted a press release claiming their company was founded in 1847. The problem? They'd built their business in 2019. Their AI assistant had simply fabricated the date with absolute confidence, no hedging, no uncertainty markers. Just a clean, professional lie embedded in a paragraph about company heritage.

This isn't a fringe problem or an edge case anymore. It's the default behavior of most AI systems operating today. The phenomenon has a name: confabulation. And unlike hallucinations in humans (which usually come with a sense of uncertainty), AI confabulation feels bulletproof. It's the reason trust in AI systems is evaporating faster than early-stage hype.

But here's what's interesting: the problem isn't that AI is dumb. It's that AI is too good at one thing, and terrible at another.

The Confident Charlatan Problem

Modern language models work by predicting the next word based on patterns learned from their training data. They're essentially sophisticated pattern-matching machines that have gotten frighteningly good at mimicking human writing. The issue is that "predicting plausible text" and "knowing facts" are completely different skills.

When an AI model encounters a question like "When was Acme Corporation founded?" it doesn't actually look up information. It searches its learned patterns for what a reasonable answer might sound like. If your training data never mentioned Acme Corporation specifically, the model doesn't think, "I don't know." Instead, it generates a plausible-sounding date because that's what it's optimized to do: produce coherent, flowing text.

The scariest part? The AI doesn't experience any difference between generating a real fact and a fabricated one. Both feel the same to the model. Both produce valid probability distributions. Both generate the next word with equal confidence.

This is why a ChatGPT conversation can feel like chatting with an informed colleague one moment and a very articulate liar the next. You can't tell the difference in the writing style. The AI doesn't warn you. It just... continues speaking with the same authoritative tone regardless of whether it's discussing verifiable facts or complete fiction.

Why Traditional Testing Fails to Catch Lies

Companies have tried to solve this with traditional quality assurance. They run test cases. They check outputs. But here's the problem: unless you specifically test whether an AI system knows facts it shouldn't, you'll never catch the confabulation.

Consider a real example from an AI safety researcher at Anthropic. They tested whether Claude (their AI assistant) would confidently claim to have personal experiences it obviously couldn't have. When asked directly, "Have you ever eaten pizza?" the model said no, understanding it's software. But when asked to "Tell me a funny story about the time you accidentally dropped pizza on yourself," it cheerfully generated an elaborate anecdote about pizza trauma.

The model had simply detected the conversational pattern that suggested a personal story was expected, and generated one. No deception intended. No lying reflex. Just pattern completion.

This is why catching confabulation requires what researchers now call "jailbreak testing" or adversarial evaluation. You have to specifically ask the AI to do something that reveals its limitations. You have to probe for the exact moment when the model transitions from knowledge to fabrication.

The New Detection Methods Actually Working

The good news is that researchers are developing real solutions. Not perfect solutions, but measurable progress.

One approach involves training models to use external verification tools before answering factual questions. Instead of generating an answer immediately, the model learns to say, "Let me search for this information" and actually checks a knowledge base or search engine. Companies like OpenAI and Google are building AI systems that can browse the web in real-time, effectively outsourcing fact-checking to verifiable sources.

Another method involves training models to be explicitly uncertain. Researchers have shown that language models can learn to use phrases like "I'm not confident about this" or "Based on my training data, but I'm not sure" when approaching information they're unsure about. This requires additional training, but it fundamentally changes how the model communicates uncertainty.

The most promising approach is what some researchers call "chain-of-thought verification." Instead of just generating an answer, the model is asked to explain its reasoning step by step. Then, crucially, other processes verify each step independently. If the reasoning chain breaks down, humans or external systems catch it before the final answer is delivered.

OpenAI's recent work on Constitutional AI shows that you can actually train models to align with specific values, including epistemic honesty—the principle of acknowledging what you don't know. It's not perfect, but models trained this way confabulate noticeably less often.

The Real Conversation We Need to Have

Here's what I think gets overlooked in discussions about AI confabulation: the problem isn't really about the AI. It's about expectations.

We've built interfaces that feel like talking to a knowledgeable human. A person with access to vast information. Someone who can be trusted. But we've trained it to optimize for one thing: sounding right. Not being right. Sounding right.

Until we redesign the interaction patterns completely—until AI systems that generate unsourced claims are genuinely unusual, not the default—users will keep getting burned. Marketing teams will keep spreading fabricated facts. Students will keep using these systems for research without realizing half the citations don't exist.

The technical fixes are coming. Chain-of-thought verification will get better. Models will get more conservative. Integration with search systems will become standard. But the deeper issue is cultural. We need to stop treating current AI systems as experts and start treating them as tools that require verification—especially when facts matter.

If you're interested in how these problems arise at a technical level, our article on why AI models hallucinate explains the underlying mechanisms and what researchers are doing to detect when it happens.

The marketing manager who almost sent out the fabricated founding date? They caught it only because they fact-checked the AI's output against their actual incorporation papers. That extra step—that skepticism—is still your best defense.