当 AI 自信地出错时，数据团队可以如何应对 — GameAnalytics

Over the past couple of weeks, we've been working through three sides of the same question: how data work in game studios is changing as AI moves into the stack. I started with the role shift itself, looking at what parts of a data analyst’s job are being automated and what the role actually becomes next. Francesco covered the architectural side in how to give an AI agent access to your game data without creating risk. And I came back last week with why dashboards aren’t enough anymore and what AI agents unlock instead.

What’s missing from all three is the question that keeps surfacing when I talk to studios that are putting these tools into production: how much do you trust the output? That’s what this piece is about.

Felix Rieseberg, who leads engineering for Claude at Anthropic, addressed this quite clearly. The relevant section of his interview is embedded below, and the rest of this piece builds on it.

The problem isn’t that AI is wrong, it's the confidence

A junior analyst’s failure mode is obvious. They hedge, second-guess, and even add caveats. The uncertainty is visible, which gives the senior analyst a chance to probe. Current-generation AI tools produce fluent, well-formatted, authoritative-sounding output whether the answer is right or wrong, and by the time someone reads it, there’s no visible signal distinguishing high-confidence correct from high-confidence hallucination. So quite the opposite from the juniors.

Felix frames the risk this way in his interview:

If the model just guesses a temperature and is correct, that should not be a passing grade. That should be bad. We should rethink what we’re doing here. But if the model is actually off by half a degree but it really figured out the right location, figured out some reputable data sources and the way it got there was reputable and good, that should actually be better than a good result.

This isn’t a bug waiting for a patch. OpenAI’s own research paper Why Language Models Hallucinate makes the structural case: current training methods reward confident guessing over honest uncertainty (actively!). A model that says “I don’t know” gets scored lower than a model that produces a plausible-sounding wrong answer. The confidence isn’t a mistake, it’s a feature of how these systems are built. Stanford HAI’s 2026 AI Index adds that AI agents fail roughly one in three structured attempts once conditions get more realistic than clean benchmark tasks, and it compounds when multiple agents interact: errors introduced by one get accepted as fact by the next.

Benn Stancil’s recent piece Can analysis ever be automated? names why this is especially hard in data work. Most software is self-validating: if the code runs, it runs. As Benn puts it, analysis is the opposite:

There’s no way to know if a chart ‘works’ unless you understand how it was built... we can only judge it by proxy: Do its conclusions sound plausible enough? Does it look like it was carefully made? Who made it?

That proxy test is exactly the test modern AI tools pass most easily. The craft markers that used to signal rigor now signal nothing, because they can be reproduced at zero cost whether the underlying analysis is correct or not. Alteryx’s 2026 State of the Data Analyst report backs this up from the practitioner side: analysts aren’t being replaced, but a meaningful share of their week now goes to validating AI output rather than producing it.

How do you catch the confident mistakes?

Most of what works here isn’t new, but data teams can no longer afford being casual about it:

Start with the rule from the interview:

Felix: I rarely want the current session to judge itself. The current Claude session tends to be quite bad at being a judge of whether or not the work it did was quite good.

The thread that produced an answer will almost always defend it, and will do so convincingly. Every AI-generated answer needs an outside signal before it’s trusted, whether that’s a separate session, a different model, or a cross-check against raw data.

The outside signal works best when it’s adversarial rather than confirmatory. Instead of asking the model if its answer is correct, spin off separate sessions whose job is to prove the output wrong. Finding flaws is cognitively easier than creating a correct answer, which is why even a weaker model, pointed at a specific artifact, often catches errors the stronger model missed. For a cohort analysis your AI tool just drafted, that means parallel sessions challenging the cohort definition, the time window, and the baseline. Adversarial review tends to surface more criticism than any one person can read, so a judge layer helps: a separate session whose only job is to decide which criticisms are substantive and which are noise.

The last principle is where to point the first three. AI isn’t trustworthy or untrustworthy as a general category. It’s trustworthy on tasks where verification is cheap and anchored to something outside the model, and untrustworthy on tasks where the output is self-contained. SQL queries that return verifiable row counts, formulas you can reproduce in a spreadsheet, and numbers you can trace to a source all sit on the trustworthy side. Interpretive analysis, strategic recommendations, and judgment calls on messy data sit on the other side, and those are the ones that need the full adversarial pipeline before anyone acts on them.

A reality check before you start

Before this all starts sounding like AI is uniquely dangerous, it’s worth remembering that the baseline isn’t a world of perfect human-generated analysis. Human analysts also present mistakes confidently (trhey just do it less fluently).

“AI can make mistakes” Bro, have you met humans?

— Leo builds (@LeoBuilds_) September 6, 2026

The practical difference isn’t that AI is less reliable than people but that is reliable at a different point in the workflow, and the verification practices that worked for catching human mistakes don’t automatically catch machine ones. The teams that figure this out early get both: the leverage of AI-assisted analysis plus the trust that comes from a verification practice that actually works. The ones that don’t get there are the ones who’ll find out, eventually, that their last three quarterly reports were built on a confident mistake nobody questioned.