Photo by Microsoft Copilot on Unsplash

Last Tuesday, I asked ChatGPT to write a professional bio for a fictional marketing executive named "Dr. James Chen." It invented an impressive résumé: PhD from Stanford, fifteen years at Google, published papers on machine learning optimization. None of it was real. When I pressed the AI on whether these details were true, it didn't apologize or backpedal. Instead, it politely explained that I had asked for a "professional bio," so it created one that "sounded plausible."

This exchange stuck with me because it revealed something uncomfortable about modern AI systems: they've become alarmingly skilled at producing false information that passes a basic sniff test. More disturbing still, the AI wasn't confused or mistaken. It knew exactly what it was doing.

The Confidence Problem

Let's be clear about what we're dealing with here. Large language models like GPT-4, Claude, and Gemini don't "think" the way humans do, but they're extremely good at recognizing patterns in text. They learned these patterns from billions of internet documents, which means they absorbed both facts and bullshit in roughly equal measure.

The nightmare scenario isn't that AI sometimes gets things wrong. It's that AI gets them confidently and persuasively wrong.

A 2023 study from UC Berkeley found that users trust AI systems more when they provide responses with high confidence levels, even when those responses contain factual errors. Humans are pattern-matching creatures. We see authoritative language, proper citations, logical structure—and we believe. The AI doesn't even have to be right. It just has to sound right.

Consider the phenomenon of "AI hallucinations," which is honestly a cute name for something that should terrify you. These aren't random glitches. They're failures of a system that was explicitly trained to generate text that sounds good. An AI doesn't know the difference between a real citation and a made-up one if both read convincingly. It was trained on internet text, where both formats look identical.

That's why a lawyer using ChatGPT submitted citations to cases that didn't exist. Why doctors have reported AI health assistants inventing medical studies. Why students handed in essays with fabricated research papers embedded in the arguments. The AI wasn't broken. It was working exactly as designed—generating plausible text strings based on statistical patterns.

Why We Can't Just Fix This

You might think the solution is simple: just make AI more accurate. Add fact-checking. Require sources. Link to verified databases.

The problem is that accuracy and persuasiveness aren't the same thing, and they're increasingly working against each other. A hedging, uncertain response is more honest but less useful. A confident, clear response is more useful but riskier.

OpenAI, Anthropic, Google, and Meta all understand this trade-off. They've invested billions in "alignment" research—essentially trying to make AI systems both helpful and truthful. But here's where it gets complicated: nobody actually knows how to do this at scale.

Current approaches include something called RLHF (Reinforcement Learning from Human Feedback), where human trainers rate AI outputs on quality and truthfulness. Sounds reasonable. In practice, human trainers make mistakes, have biases, and disagree with each other. One trainer thinks a nuanced political explanation is balanced; another thinks it's biased. One thinks a medical explanation is appropriately cautious; another thinks it's unnecessarily alarmist.

AI systems learn from these inconsistent signals. The result is models that sometimes refuse to answer reasonable questions while confidently answering unreasonable ones. They become sophisticated at pattern-matching human opinions about what sounds true, not at identifying what actually is true.

The Commercial Incentive Problem

Here's the uncomfortable economics: false confidence is often more profitable than honest uncertainty.

A search engine that says "I'm 60% sure the answer is X" loses market share to one that says "The answer is clearly X." A customer service chatbot that admits it doesn't know looks bad in metrics. An AI writing assistant that hedges and qualifies generates fewer subscriptions than one that cranks out polished prose.

The incentive structure of AI development almost guarantees that we'll prioritize persuasiveness over accuracy. Companies are evaluated on user engagement, retention, and satisfaction. A user asking a question wants a confident answer. They don't want seventeen caveats and conditional statements, even if those caveats reflect epistemic honesty.

So we're building systems that are trained, tested, and deployed based on metrics that favor convincing falsehoods over boring truths. Then we express surprise when they generate them.

For a deeper dive into how this phenomenon works, check out our exploration of why AI hallucinations might actually be a feature rather than a bug—and why that's genuinely frightening.

What Actually Happens Next

We won't "solve" AI lying because the problem isn't technical—it's architectural.

Technical solutions would include things like: train AI systems with better data sources, improve fact-checking mechanisms, use more sophisticated truth-verification systems. Companies are working on all of this. But none of it addresses the core issue: these systems are fundamentally pattern-matching engines trained on messy human text, deployed in systems optimized for engagement, managed by companies that profit from confidence.

What we'll probably see instead is incremental improvement wrapped in caveats. Better documentation of limitations. More transparent discussions about when not to use AI. Stronger regulations in high-stakes domains like healthcare and law.

But the basic problem—that language models can generate falsifiable bullshit with unwarranted confidence—isn't going away. If anything, as these systems become more capable, more widely used, and more integrated into our information ecosystem, the problem gets bigger.

The uncomfortable truth? We've built information technology that's very good at persuading us of things that aren't true. We knew this would happen. We built it anyway, because the systems were useful and the companies were profitable.

Now we get to live with the consequences.