Photo by Microsoft Copilot on Unsplash

Last month, a lawyer in New York filed a brief citing six legal cases that don't exist. He didn't make them up. An AI chatbot did. The tool had generated the citations with absolute conviction, complete citations and all, even though the cases existed nowhere in any law library. The lawyer trusted it. The judge was not amused.

This wasn't a glitch. It was a feature.

The Confidence Problem Nobody Designed For

When you ask an AI system a question, you're not really asking a thinking being to retrieve information. You're asking a statistical pattern-matcher trained on billions of words to predict what comes next. The problem? These systems have no internal mechanism for uncertainty. They can't say "I don't know." They can only say what comes next in the sequence, which often sounds absolutely credible.

OpenAI's own researchers have documented this extensively. A model trained on internet text will compress entire fields of human knowledge into mathematical weights, then reconstruct reasonable-sounding answers from those patterns. When you ask about medieval history, quantum physics, or tax law, the model doesn't consult a database. It guesses—but it guesses in the most confident possible voice.

Consider what happened when researchers at UC Berkeley asked GPT-3 questions about basic facts. The model would answer questions like "Is Rome the capital of France?" with paragraphs of supporting argument, completely fabricated context, and total conviction. It didn't hedge. It didn't say "I think." It presented falsehoods as established facts, sometimes even citing sources that don't exist.

The Sourdough Problem: Why This Is Actually Worse Than It Sounds

The real issue isn't that AI makes mistakes. Humans make mistakes constantly. The issue is that AI makes mistakes while sounding like an authority. This creates a specific type of problem that bleeds into everything from medical advice to investment guidance.

Think about how you interact with information normally. When you read an academic paper, you check the credentials. When a friend gives you advice, you consider whether they know what they're talking about. When a stranger on the internet makes a claim, you're skeptical. But when an AI system answers a question? Most people treat it like gospel.

There's actually an entire phenomenon researchers call "confabulation" in large language models. The system generates plausible-sounding text that fills in gaps where it has no actual knowledge. It's not lying intentionally. It's doing exactly what it was trained to do: predict the next word in a sequence.

The kicker? As these models get larger and more sophisticated, they get better at this. They're not improving at being right—they're improving at sounding right. A 2023 study found that as language models scaled up, human evaluators actually became less able to distinguish between correct and incorrect answers, even though the error rate hadn't necessarily improved. The systems are learning to sound more authoritative, not more accurate.

What Happens When Institutions Depend on This?

We're at a genuinely strange inflection point. Organizations are starting to depend on AI systems for everything from customer service to content generation to—believe it or not—scientific research. And they're doing this without adequate safeguards for the confidence problem.

A marketing agency might use AI to write product descriptions. A half-dozen descriptions sound great, are on-brand, and absolutely convince customers to buy something. But what if those descriptions contain false claims about the product's capabilities? The customer bought it based on fabricated specifications.

Medical researchers have started finding AI-generated "studies" in citation databases. Not because the papers were peer-reviewed and published, but because AI systems had generated them based on training data, and other AI systems had cited them as real. The error propagated through the system.

This isn't science fiction. It's happening right now. The New England Journal of Medicine has had to issue guidance on detecting AI-authored papers because researchers are submitting them. They look legitimate. They cite real research. They're structured correctly. They're also often completely fabricated.

The Path Forward: Uncertainty as a Feature

Some researchers are working on this problem. One approach involves training models to express uncertainty explicitly. Instead of always sounding confident, a model could be trained to flag statements it's less sure about. Could output probability scores. Could say "I'm less confident about this" for novel or edge-case information.

Other teams are building verification systems—AI trained specifically to check other AI's work, to catch hallucinations and flag confident falsehoods. It's not perfect, but it's something.

The real shift needs to be cultural, though. Organizations using AI systems need to treat them as tools that generate hypotheses, not provide answers. A lawyer shouldn't use an AI-generated legal citation without verifying it. A doctor shouldn't rely on AI diagnostic suggestions without independent confirmation. A researcher shouldn't submit an AI-written paper without reviewing every claim.

That sounds obvious, but it's hard to do consistently. Because these systems are so good at sounding right.

The Uncomfortable Truth

The uncomfortable truth is that we've built systems that are incredibly skilled at appearing authoritative while operating with no actual understanding of what they're saying. They don't know they don't know. They can't know. That's not how they work.

And as they get better, smarter, and more integrated into how we work, we need to get better at treating them with appropriate skepticism. Not because they're evil. But because they're not conscious entities with internal maps of the world. They're statistical engines. Powerful ones. But engines nonetheless.

The New York lawyer learned this the hard way. Hopefully, many others can learn it without the judge's intervention.