Photo by Luke Jones on Unsplash

Last month, I watched a language model confidently explain the chemical composition of a fictional element with the same authority it used to describe water. The person using it believed every word. That's not a bug—that's the feature that's quietly reshaping how millions of people interact with AI.

The problem isn't that AI is getting dumber. It's that AI is learning to sound smarter, regardless of whether it actually knows anything.

The Confidence Paradox Nobody Expected

When engineers first trained large language models, they discovered something bizarre: the models' ability to sound authoritative had almost nothing to do with accuracy. A model trained on billions of words learns patterns. It learns that confident statements followed by specific details tend to appear frequently in its training data. So it replicates that pattern.

Think about how humans learn confidence. You practice something repeatedly, you gain experience, your brain literally rewires itself. With AI, there's no experience. There's just pattern matching. The model has never actually done anything—it's just seen thousands of examples of how confident people write about doing things.

This creates what I call the confidence loop. The model generates text that sounds authoritative. That text gets used by real people in real situations. Those people then treat it as reliable information. That information sometimes gets fed back into training data for the next generation of models. The cycle strengthens. The confidence deepens. The accuracy often stays exactly where it started—mediocre.

Why This Matters More Than You Think

Consider what happened in early 2024 when a lawyer filed legal briefs citing cases that didn't exist, all because ChatGPT confidently invented them. The model didn't know it was wrong. It had simply learned to generate text that followed the pattern of authentic legal citations. The lawyer didn't know it was wrong because the format was perfect.

Or look at medical students using AI to study. Research shows they sometimes perform worse after using AI assistants, not because the AI actively teaches them falsehoods, but because the AI presents guesses with the same confidence it presents facts. The student's brain doesn't distinguish between the two.

This isn't a glitch in one particular model. This is a fundamental consequence of how these systems work. You can't train a system on human-generated text and expect it to mysteriously know which patterns represent truth and which represent convincing lies. The problem runs deeper than most people realize—it's baked into the training process itself.

The Scaling Problem That Gets Worse, Not Better

Here's where it gets genuinely uncomfortable: larger models often get more confident, not less. Counterintuitive, right?

A smaller model might hedge its bets. It might say "I'm not entirely sure, but..." A larger model has been trained on more diverse text, including more examples of confident statements, so it learns to mimic that confidence at scale. More parameters don't equal more accuracy—they equal more sophisticated pattern matching.

OpenAI, Anthropic, Google, and Meta are all investing billions into making bigger models. The assumption is that scale solves problems. But what if scale exacerbates this particular one? What if we're building systems that are simultaneously more capable and more confidently wrong?

Some labs are experimenting with uncertainty quantification—trying to make models express doubt. But that's fighting against the grain of how these systems fundamentally work. You can't easily train confidence out of a system built to predict the next token based on patterns in human writing, where confident people write constantly.

What Actually Happens When We Ignore This

The real danger isn't the dramatic scenarios—the AI lawyer or the dangerous medical advice. Those make headlines, so they get caught. The danger is the slow, systematic erosion of epistemic standards.

When millions of people use AI daily to write emails, generate code, brainstorm ideas, and research topics, they're being constantly exposed to confident assertions that range from perfectly accurate to completely fabricated. Over time, this trains human brains to be less discerning about sources. It trains people to trust the format of information rather than its origin.

We're simultaneously building systems that can't reliably know what they don't know, while teaching humans to expect instant answers formatted with authority. That's not a technology problem. That's a civilization problem.

The Uncomfortable Path Forward

So what do we actually do about this? The honest answer is: we don't have a great solution yet, and most people building these systems seem reluctant to admit it.

Some ideas are being tested: having models explicitly show their reasoning, training them to express uncertainty, limiting their use in high-stakes domains. None of these are perfect. They all have tradeoffs.

The uncomfortable truth is that we've built systems that are genuinely useful and genuinely dangerous in ways we didn't anticipate. The usefulness makes it hard to restrict them. The danger makes it hard to ignore them.

What we absolutely cannot do is pretend this is solved. We cannot keep deploying increasingly confident AI systems while researchers quietly publish papers about their limitations. We cannot let the confidence loop accelerate unchecked while treating it as a known issue we're "working on."

The system isn't broken. It's working exactly as designed. That's precisely the problem.