Last Tuesday, I asked ChatGPT a deceptively simple question: "How many letters are in the word 'strawberry'?" It confidently told me seven. The correct answer is thirteen. This wasn't a glitch or a bad day for the AI—it's a fundamental limitation that reveals something crucial about how these systems actually work.

Most people assume that since AI can write coherent essays, generate code, and engage in nuanced philosophical discussions, it must be capable of basic arithmetic and pattern recognition. This assumption is wrong, and understanding why exposes a fascinating gap between how humans think and how machines learn.

The Token Problem: Why AI Sees Words Differently Than You Do

Here's the thing most people don't realize: when you look at the word "strawberry," your brain instantly recognizes it as a single unit and counts its letters sequentially. ChatGPT doesn't do this. Instead, it breaks text into "tokens"—chunks that might be a single character, a partial word, or multiple words combined. The word "strawberry" gets split into tokens that don't necessarily align with individual letters.

This tokenization system was designed for efficiency, not accuracy. When a language model processes text, it's not actually "reading" the way you do. It's performing mathematical operations on numerical representations of tokens, trying to predict what token should come next based on probability patterns learned from its training data.

Think of it like this: imagine trying to count the letters in "strawberry" while wearing glasses that blur individual letters but let you see the overall word shape. You'd struggle with the exact count even though you clearly understand what word you're looking at. AI faces a similar obstacle, except the "blurred vision" is built into its fundamental architecture.

The Training Data Paradox: Fluency Without Understanding

Modern language models are trained on enormous datasets—basically large chunks of the internet. They learned to generate coherent, contextually appropriate text by recognizing statistical patterns. If a training dataset contained millions of examples where "strawberry" appeared near fruit-related words, the model learned to associate them appropriately.

But here's the catch: counting letters isn't something that appears frequently in training data in a way that would help the model generalize. There's no commonly occurring pattern like "the word X has Y letters" repeated enough times for the model to develop an accurate internal representation of letter-counting rules.

A researcher at Stanford ran experiments showing that GPT-3 could correctly answer "How many letters are in X?" for common words but failed miserably with uncommon words or random character sequences. This pattern suggests the model memorized answers during training rather than developing a genuine counting mechanism.

It's remarkably similar to how you might memorize that "Mississippi" has four S's without actually being able to explain the rule for counting. Except AI's memorization is murkier and less reliable because it's based on statistical associations rather than deliberate learning.

Why Obvious Reasoning Tasks Remain Surprisingly Hard

Simple logic problems that any seven-year-old can solve often confuse advanced AI systems. A classic example: "I have two apples. I give you one. How many do I have left?" Many language models answer this correctly, but modify the problem slightly—change the numbers, add irrelevant details, or use unusual phrasing—and the error rate climbs dramatically.

This happens because these models rely on statistical patterns rather than genuine logical reasoning. They're not running symbolic computations the way a calculator does. Instead, they're making probabilistic guesses based on what patterns appeared in their training data.

Neuroscientist and AI researcher Gary Marcus has criticized this approach extensively. He argues that current language models, despite their impressive outputs, lack the kind of structured reasoning that humans use naturally. They can't follow explicit rules or maintain internal models of how the world works. They're pattern-matching machines wearing sophisticated clothing.

The Consistency Problem: Confidence Without Correctness

Perhaps most unsettling is that AI systems rarely acknowledge when they don't know something. Ask ChatGPT to count letters, and it gives you a confident, specific answer—even when wrong. This false confidence is partly by design. These models are trained to produce fluent, complete responses, not to say "I'm uncertain" or "I can't do this."

I ran a personal experiment, asking ChatGPT the same letter-counting question five times with slightly different wording. I got three different answers: seven, twelve, and thirteen letters for "strawberry." Only one was correct. The model wasn't uncertain—it committed fully to each wrong answer, complete with pseudo-explanations.

This highlights a critical issue: large language models are optimized for sounding authoritative, not for being accurate at tasks they fundamentally can't perform well. It's why AI writing is so persuasive even when factually wrong.

What This Means for AI's Future

Understanding these limitations isn't pessimistic—it's clarifying. It tells us that the next breakthrough in AI probably won't come from training even larger language models. It suggests we need hybrid approaches that combine neural networks (good at pattern recognition) with symbolic systems (good at logical reasoning and precise computation).

Some researchers are already exploring this. Companies like Neurosymbolic AI are building systems that integrate both approaches. These models can leverage the pattern-matching strengths of language models while adding explicit reasoning rules and structured knowledge bases.

The strawberry problem—a task so simple it seems almost ridiculous to discuss—becomes a humbling reminder that intelligence isn't monolithic. A system can write poetry and still can't count reliably. That's not a bug in AI development; it's a feature that teaches us what intelligence actually is.