I Let AI Generate My Training Data. It Poisoned Everything.

Six months.

That’s how long my prediction engine was training on corrupted data before I noticed. Six months of calibration runs. Six months of accuracy reports that looked fine. Six months of me thinking the system was getting smarter when it was actually getting more confident about wrong answers.

And the worst part: I did it to myself.

The shortcut

Scoring products by hand is slow. Each case takes an hour of research, careful analysis, honest evaluation. I had about 200 hand-scored cases. I needed more.

So I did the obvious thing. I gave an AI my scoring rubric, showed it examples of my hand-scored work, and asked it to generate new cases.

The output was impressive. Scores within valid ranges. Analysis that was coherent. Reasoning that made sense. Relative rankings that looked right.

I added 400 AI-generated cases to my dataset and moved on to other problems.

The moment I knew

Six months later, I was debugging something unrelated — late at night, two browser tabs open side by side — and decided to spot-check a few AI-generated scores against products I’d also scored by hand. “Just a quick sanity check,” I told myself.

User frustration with existing products. I’d scored it 50 for a particular case. The AI had scored it 70.

I checked another. I’d said 45. The AI said 68.

Another. Mine: 52. AI: 74.

I checked twelve more. Every single one was inflated. The AI was scoring user frustration 20 points higher than me. Consistently. Across every product in the dataset.

Then I checked market momentum. Same pattern. The AI had it 8 to 20 points higher than my hand-scored values. Every time.

Everything else was fine. Five other features — all within normal variance of my hand scores. Just these two. But these two were systematically wrong in the same direction for six months.

Why AI does this

Think about what the internet sounds like.

“I HATE this product.” “This market is EXPLODING.” “Users are FURIOUS with the status quo.”

That’s what gets shared. That’s what gets clicks. “Zoom is adequate for my needs” doesn’t go viral. “The CRM market grew 3% this year” doesn’t get written.

AI trained on millions of these articles. It absorbed the drama and started treating it as normal. So when I asked “how frustrated are users with the current solution?” it defaulted to “very” — because that’s what the internet always says.

It wasn’t lying. It was reflecting the world it was trained on. And that world is louder, angrier, and more dramatic than reality.

Why every check I had missed it

This is the part that keeps me up at night.

Scores within valid range? Yes — the AI didn’t score anything at 150 out of 100. The numbers were plausible.

Distribution looked reasonable? Yes — there was variance. Not everything maxed out.

Written analysis supported the scores? Yes — the reasoning was coherent. The AI built a good argument for why frustration was 70, not 50.

Rankings between products made sense? Mostly yes — the relative order was fine. If Product A was more frustrating than Product B in my hand scores, the AI agreed. It just had both 20 points higher.

A score of 70 instead of 50 doesn’t look obviously wrong on a 0-100 scale. It’s within the range where two reasonable humans might disagree. The bias was small enough to hide inside the normal range of disagreement. You’d only catch it by lining up hundreds of cases against hand-scored ground truth.

Which I didn’t do. For six months.

The fix

Ugly. Simple. Effective.

I computed how much the AI inflated each score compared to my hand-scored data and applied a correction factor. User frustration: multiply every AI score by 0.67. Market momentum: multiply by 0.87.

The moment I applied these corrections, my prediction engine’s accuracy jumped. Not because the model got smarter — because it was finally working with honest data instead of inflated data.

Why we trust the machine

This is the question I keep coming back to.

I had six months of warning signs I could have caught. I could have spot-checked AI scores against my hand scores on day one. I could have compared distributions between the two datasets on week one. I could have built a drift detector that flagged when AI scores consistently skewed from human scores.

I didn’t. Because the data passed my automated checks and the outputs looked professional. Clean formatting and confident reasoning made me trust outputs I should have verified.

We do this everywhere. A polished slide deck makes us trust the numbers in it. A well-dressed person makes us trust their advice. A coherent paragraph makes us trust the facts it contains. Our brains mix up looks good with is right — and AI hits this blind spot without even trying.

What I do now

Every new AI-generated case gets automatically adjusted using my correction factors. I spot-check 10 AI scores against hand scores every week, keep a running comparison, and update the corrections when they drift. Computing the correction is simple: divide your hand-scored average by the AI’s average for each feature.

It’s more work than trusting the AI output at face value. But it’s a lot less work than discovering six months later that your entire system has been slowly poisoned by data that looked perfect and was quietly wrong.

You can’t validate AI data with more AI. The biases are consistent. If the AI inflates frustration for one product, it inflates it for all of them.

Comparing AI against AI is like asking two people who read the same newspaper to fact-check each other. (This is why I eventually built a separate verifier instead of trusting the generator to check itself.) They’ll agree on everything — including the parts that are wrong.

Trust your ground truth. Not your AI’s opinion of itself.

---

Using AI to generate training data? I’ve already stepped on this landmine so you don’t have to — mo@fadaly.net.