Photo by Luke Jones on Unsplash

Last year, a team at OpenAI noticed something unsettling. They trained a language model on an expanded dataset—more examples, better curated information, the works. By every metric, it should have been an upgrade. Instead, performance tanked on specific tasks. The model actually got worse at understanding certain types of instructions, even though it had been trained on more examples of those exact instructions.

This isn't a one-off bug. It's a real phenomenon that's starting to shape how companies approach AI development, and it reveals something genuinely strange about how these systems learn.

The Intuitive Expectation That Keeps Failing

Our brains have a simple rule: more practice makes you better. If you practice basketball shots a thousand times, you're better than if you practice a hundred times. This assumption feels so obviously true that we've built entire educational and professional development systems around it.

AI researchers made the same assumption. Feed a model more data, and it learns better patterns. Scale up, and scale up more. For years, this worked reliably. Bigger models trained on bigger datasets consistently beat their smaller cousins. It became the unwritten law of machine learning: bigger is better, period.

But reality is messier than our intuitions allow. Sometimes, adding more data actually introduces noise that confuses the model. Sometimes expanding a dataset means including contradictory examples that pull the model in opposite directions simultaneously. And sometimes—this is the weird part—the model learns spurious correlations from the larger dataset that didn't exist in the smaller one, causing it to make worse predictions on novel cases.

When Quantity Becomes the Enemy of Quality

Imagine training an AI to recognize whether a photo contains a dog. Your original dataset has 10,000 carefully labeled images: 5,000 with dogs, 5,000 without. The model learns real features—fur texture, ear shape, body proportions. It performs well.

Now you expand the dataset to 100,000 images by scraping the internet and using cheaper labeling methods. You get more coverage, more diversity. Sounds great, right? Except now your data includes mislabeled images, watermarks that appear on dog photos more often than non-dog photos, and weird correlations (maybe dog photos tend to be taken outdoors more often, so the model learns to associate "grass" with "dog").

The model doesn't distinguish between signal and noise the way a human would. It finds patterns everywhere, even in the artifacts and errors. The result: it performs worse on clean, real-world test data because it's been trained to recognize patterns that don't actually matter.

This phenomenon, sometimes called "dataset contamination" or the "more data curse," has shown up repeatedly in recent research. A 2023 study from Stanford and Google found that certain image classification models trained on larger, noisier datasets actually performed worse on standard benchmarks than models trained on smaller, carefully curated ones.

The Forgetting Curve Nobody Expected

There's another mechanism at play here that's even more unsettling. AI models have something researchers call "catastrophic forgetting." When you train a model on task A, then train it on task B, it can literally forget how to do task A.

This happens at the data level too. If your expanded dataset is imbalanced—say you add a million new examples of one type and only a thousand of another—the model's attention shifts. It becomes an expert on the abundant class and forgets the subtleties of the sparse one. The model hasn't become dumber in an absolute sense, but it's become dumber at the specific things you cared about.

A practical example: you're building a spam classifier. Your original data was balanced—equal spam and legitimate emails. You expand it by scraping the web, but now you have 100,000 examples of legitimate emails and only 5,000 of spam. The model learns to be very confident about what "legitimate" looks like and essentially gives up on understanding spam patterns. When new, creative spam arrives, it fails spectacularly.

What Teams Are Actually Doing About This

Smart AI teams have started treating data curation like it's more important than model architecture. Google's recent work on "data-centric AI" is a perfect example. Instead of asking "how do we build a bigger, better model?", they're asking "how do we build a better dataset?"

This means spending engineering resources on data cleaning, removing redundant examples, identifying and correcting labeling errors, and thoughtfully balancing classes. It's less glamorous than inventing new neural network architectures, but it actually works.

Some teams are also experimenting with "selective data augmentation"—carefully choosing which additional data to add rather than blindly adding everything. Others use weighted training approaches where the model learns to ignore or downweight examples that seem noisy or contradictory.

There's also a connection here to a broader challenge we've written about before. AI systems sometimes perform worse when scaled up in unexpected ways, and understanding the mechanisms behind that degradation is becoming just as important as understanding how to scale effectively.

The Uncomfortable Truth

The bigger lesson here is humbling. We built AI systems based on assumptions that felt obviously correct. More data equals better performance. Larger models equal smarter systems. These assumptions held true for long enough that we built them into the foundation of modern AI development.

But the real world is full of edge cases where our intuitions break down. Sometimes less is more. Sometimes constraints force you to think harder about what actually matters. Sometimes the path to smarter AI isn't to feed it more information—it's to feed it better information.

As AI becomes more central to real-world systems—medical diagnosis, hiring, financial decisions—getting this right matters enormously. A chatbot that hallucinates is annoying. An AI system that gets worse as you give it more data to learn from? That's genuinely dangerous.

The field is slowly learning this lesson, but the learning curve is steep. And unlike AI models, humans seem to need more than just more examples to really change our minds.