How to tell whether an AI feature actually works

Your AI feature worked when you demoed it. That is the only evidence most teams collect before shipping.

In August 2026 Microsoft published a benchmark that measures the thing that demo misses. They ran 507 real business workflows through AI agents, across retail, hospitality, auto insurance, banking IT and consulting support. Every workflow was attempted 20 times, and each attempt was graded on what actually changed in the backend database, not on what the agent said it did.

The strongest model completed 65.36% of workflows on the first try. It completed all twenty attempts on only 25.25% of them.

Their paper is titled "One Success Isn't Reliability."

Testers have a shorter word for the gap between those two numbers. We call it a flaky test: one that passes and fails on the same code, with nothing changed but time.

This post is the method for measuring it in your own product.

Why one good run tells you almost nothing

A traditional test is deterministic. Given the same input and the same code, it returns the same answer every time. When it fails, something is broken.

An AI feature is not. The same prompt against the same model can return a different answer on Tuesday than it did on Monday. Temperature, model updates, retrieval order, and load all move the output.

So "it worked" is not a property of the feature. It is a property of one attempt. The question worth answering is: how often, and how far does it drift when it misses?

The ten-run check

Here is the smallest honest version. Drop it in a spec file, replace score() with whatever "good" means for your feature, and run it against the AI path you trust least.

import { test, expect } from '@playwright/test'

const RUNS = 10
const MIN_PASS_RATE = 0.9   // 9 of 10 runs must be good enough
const MAX_SPREAD = 0.34     // best run minus worst run

test('the summary feature holds up over ten runs', async ({ page }) => {
  const scores: number[] = []

  for (let i = 0; i < RUNS; i++) {
    await page.goto('/summarize')
    await page.getByRole('button', { name: 'Summarize' }).click()
    const text = await page.getByTestId('summary').innerText()
    scores.push(score(text))
  }

  const passRate = scores.filter(s => s >= 0.67).length / RUNS
  const spread = Math.max(...scores) - Math.min(...scores)
  console.log(`pass rate ${passRate}  spread ${spread.toFixed(2)}  [${scores}]`)

  expect(passRate).toBeGreaterThanOrEqual(MIN_PASS_RATE)
  expect(spread).toBeLessThanOrEqual(MAX_SPREAD)
})

// Yours to write. Start with the cheapest checks that are not "did it crash".
function score(text: string) {
  let s = 0
  if (text.length > 40) s += 0.34                  // it said something
  if (/\d{4}-\d{2}-\d{2}/.test(text)) s += 0.33    // it kept the date
  if (!/i cannot|as an ai/i.test(text)) s += 0.33  // it did not refuse
  return s
}

Two things make this useful instead of noisy.

Log all ten scores every run, and keep them. One bad day means nothing. The same feature drifting down across a week is the real signal.

Fail on the spread, not only on the average. A feature scoring 1.0, 1.0, 0.33, 1.0 has a worse problem than one scoring 0.8 every time. The average hides it. The spread does not.

The flag you already have

If you would rather not write a loop, Playwright ships this:

npx playwright test --repeat-each=10

It runs the same test ten times, and every run has to pass.

That tells you whether something failed. It does not tell you how far the answers moved, which is the part that warns you early. Use it as the free first step, then graduate to scoring when the feature matters.

The setting that hides all of this

Most projects carry this in playwright.config.ts:

export default defineConfig({
  retries: 2,
})

A failed test gets two more chances. If any of them passes, the run reports the test as flaky and still exits 0. The build goes green and nobody opens it.

Retries exist for a good reason: they keep a shared pipeline usable. But they are the opposite tool from the one above. Retries let one pass overwrite two failures. Repetition refuses to let that happen.

You need both, and most teams only have the first.

What to do this week

1. Pick the three AI-backed paths you trust least.
2. Run each ten times and write down the pass rate and the spread.
3. Decide, before you see the numbers, what pass rate would stop a release.
4. Put that number in the test, not in a document.

Step 3 is the one teams skip, and it is the one that makes the rest mean anything. A threshold chosen after seeing the result is not a threshold.

The part worth keeping

Microsoft needed 507 workflows, twenty runs each, and a graded backend state to say something testers have said for twenty years: one success is not reliability.

The measurement is not new. What is new is that the software now behaves differently on Tuesday than it did on Monday, so everyone needs the measurement, not just the people with "quality" in their title.

---

Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.