Published: · 4 min read
Use seven plain checks to decide whether an AI-generated test proves a real user risk and deserves a place in your test suite.
Review AI-generated tests with seven checks. Name the user risk. Break the product. Check the final result. Change the data. Read the failure. Repeat the run. Then decide whether the test could stop a bad release.
AI can write a clean test in seconds. The file may look finished. The names may sound correct. The test may even pass.
None of those facts prove value. A useful test catches a failure that matters to users. Your review must find that proof before the test joins your suite.
I use seven checks for that decision.
Start with the person who could get hurt. Write the risk in one sentence.
For example: "A customer sees the wrong total and pays too much."
Avoid risks like "checkout may fail." That sentence does not name the damage. It also gives the test no clear target.
Ask two questions:
Money, time, access, and trust are clear answers. "The feature breaks" is not.
A test should fail when its protected behavior breaks. Prove that before keeping it.
AI often writes a test that follows the correct steps. The test may never check the important result.
test('customer can pay', async ({ page }) => {
await page.goto('/checkout')
await page.getByRole('button', { name: 'Pay' }).click()
await expect(page.getByText('Success')).toBeVisible()
})
This test checks a message. It does not check the charged amount.
Remove the payment action or return the wrong total. The test must fail. A passing result means the test protects nothing useful.
Clicks are steps. Results are proof.
The test above clicks the right button. A stronger test checks the amount and payment record.
await expect(page.getByTestId('order-total')).toHaveText('$120.00')
await expect(page.getByTestId('payment-status')).toHaveText('Paid')
An assertion means a result check. Playwright provides assertions that wait for results. The tool can wait. You still choose the right result.
Review every assertion. Ask what user outcome it proves. Rewrite assertions that only confirm page activity.
AI tends to generate a clean example. Real users bring missing, wrong, and extreme values.
Test more than one amount. Include zero, a large value, and invalid text.
for (const amount of ['0', '999999', 'wrong']) {
await page.getByLabel('Amount').fill(amount)
await page.getByRole('button', { name: 'Pay' }).click()
await expect(page.getByRole('alert')).toBeVisible()
}
The exact values depend on your product. The review question stays simple. Can one neat example hide a serious failure?
Run the test against a broken result. Then read its message.
Another engineer should understand the problem without opening the whole file. Compare these messages:
Expected: "$20.00"
Received: "$120.00"
Timeout after 30000ms
The first message points to the product error. The second sends someone searching through logs.
Use clear result checks and useful test names. A failure should shorten the investigation.
Use the same input twice. You should get the same result.
Repeated runs catch shared data and timing problems. They also expose tests that depend on another test.
Do not accept a passing second run as proof. Compare both runs. Investigate any difference before keeping the test.
Finish with one question: Would this test stop a bad release?
Name the release it could stop. For example: "This test blocks checkout when totals are wrong."
Keep the test when the answer is clear. Rewrite or delete it when the answer stays vague.
This step protects the review queue. Teams do not need every generated test. They need the small set that proves important behavior.
Imagine that AI writes a checkout test. The test adds one product and completes payment. It checks the success message.
First, name the risk. A customer could pay the wrong total.
Next, change the price calculation. The test still passes because the message appears. You found the missing proof.
Add a check for the order total. Then try an expired coupon and an empty cart. The test should explain each failed result.
Run the test twice with the same data. Both runs should match.
Finish with the release question. This test should stop checkout when the charged total is wrong. The answer now names one clear release failure.
That review improved one test without adding more code than needed. The team gained useful proof instead of another passing file.
You can complete this review in a few minutes:
AI can handle test volume. You still decide what deserves trust. Keep tests that fail correctly and explain why the release should stop.
Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.
Get notified when I publish something new, and unsubscribe at any time.
·4 min read
A test that fails, then passes on retry, is not fixed. Two readers explain why, with the race-condition case and a Playwright setup that treats retries as detection.
·5 min read
Your login helper exists twice: one that expects success, one that expects an error. Here is the options-object pattern that keeps one method, with Playwright code.
·4 min read
Should page objects contain assertions? A practical rule: business checks live in tests, technical guards live in page objects. With Playwright code.
