Anton Gulin — Test Automation Blog

Your Regression Suite Is a Museum: 5 Questions That Decide Delete vs Keep

Anton Gulin — Wed, 19 Aug 2026 10:00:00 GMT

Your Regression Suite Is a Museum: 5 Questions That Decide Delete vs Keep

Every sprint adds tests. Almost no sprint deletes one.

That one-way flow has a predictable end. The suite gets slower. Flaky tests
(tests that fail randomly) pile up. Engineers stop trusting red builds.
Eventually someone suggests rewriting everything, and the cycle restarts.

There is a cheaper fix. Audit the suite with five questions.
On the last suite I inherited, they retired a third of the tests.
Nothing those tests "guarded" ever broke.

Question 1: When did this test last fail for a real reason?

Your CI history (the record of server test runs) already knows.

A test that has been green for a year has two possible explanations.
Either the code it guards never changed, or the test cannot detect change.
Both are worth knowing. Only one deserves compute on every commit.

Practical rule: pull the last 90 days of runs. Tag every test that failed
only for environment reasons, or never failed at all. Those are audit candidates.

Question 2: What user risk does it guard?

Write the risk in one sentence, naming a user.

"It tests the profile page" is not a risk.
"A user loses saved work when the session expires" is a risk.

If nobody on the team can produce that sentence, the test is guarding
an implementation detail, not a user. Details change on purpose all the time.
Tests guarding them fail on purpose all the time. That is where flakiness lives.

Question 3: Would anyone notice if it vanished?

Run the experiment instead of debating it:

# in a branch: skip the suspect test, run everything else
npx playwright test --grep-invert "@museum-candidate"

If coverage of the named risk survives through other tests, and no gap
appears in the risk list from Question 2, the test was furniture.

Delete it in the branch. Keep the branch open a week. Merge with confidence.

Question 4: Does it check results or steps?

A test that clicks through checkout and asserts the button was clickable
is a tour, not a test.

// a tour: asserts the step happened
await page.getByRole('button', { name: 'Pay' }).click();
await expect(page).toHaveURL(/confirmation/);

// a test: asserts the outcome is real
await expect(page.getByTestId('invoice-total')).toHaveText('$34.20');
await expect(page.getByTestId('invoice-number')).not.toBeEmpty();

The URL can change while the invoice ships blank. Assert on the thing
the user came for.

Question 5: Can it fail at all?

Flip the condition and run it.

// original
await expect(status).toBe('paid');
// flipped: this MUST fail. If it passes, the test is dead.
await expect(status).not.toBe('paid');

A test that passes both ways asserts nothing. AI-written tests fail this
check more than any other kind, because generators optimize for green.

Sixty seconds per suspicious test. The flip never lies.

What you get back

Five questions, one afternoon, on the oldest third of your suite.

The suite you keep is faster, and every test in it can answer
"what breaks if I go red?" That is what makes a red build mean something.

And when an AI agent starts writing tests into your suite, this audit
is the contract you hold its work against. A generated test enters
only if it survives the same five questions.

Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

How to Review AI-Generated Tests: Seven Checks Before You Keep Them

Anton Gulin — Wed, 12 Aug 2026 10:00:00 GMT

Post

Review AI-generated tests with seven checks. Name the user risk. Break the product. Check the final result. Change the data. Read the failure. Repeat the run. Then decide whether the test could stop a bad release.

AI can write a clean test in seconds. The file may look finished. The names may sound correct. The test may even pass.

None of those facts prove value. A useful test catches a failure that matters to users. Your review must find that proof before the test joins your suite.

I use seven checks for that decision.

1. Name the user risk

Start with the person who could get hurt. Write the risk in one sentence.

For example: "A customer sees the wrong total and pays too much."

Avoid risks like "checkout may fail." That sentence does not name the damage. It also gives the test no clear target.

Ask two questions:

Who loses something?
What do they lose?

Money, time, access, and trust are clear answers. "The feature breaks" is not.

2. Break the product

A test should fail when its protected behavior breaks. Prove that before keeping it.

AI often writes a test that follows the correct steps. The test may never check the important result.

test('customer can pay', async ({ page }) => {
  await page.goto('/checkout')
  await page.getByRole('button', { name: 'Pay' }).click()
  await expect(page.getByText('Success')).toBeVisible()
})

This test checks a message. It does not check the charged amount.

Remove the payment action or return the wrong total. The test must fail. A passing result means the test protects nothing useful.

3. Check the final result

Clicks are steps. Results are proof.

The test above clicks the right button. A stronger test checks the amount and payment record.

await expect(page.getByTestId('order-total')).toHaveText('$120.00')
await expect(page.getByTestId('payment-status')).toHaveText('Paid')

An assertion means a result check. Playwright provides assertions that wait for results. The tool can wait. You still choose the right result.

Review every assertion. Ask what user outcome it proves. Rewrite assertions that only confirm page activity.

4. Change the data

AI tends to generate a clean example. Real users bring missing, wrong, and extreme values.

Test more than one amount. Include zero, a large value, and invalid text.

for (const amount of ['0', '999999', 'wrong']) {
  await page.getByLabel('Amount').fill(amount)
  await page.getByRole('button', { name: 'Pay' }).click()
  await expect(page.getByRole('alert')).toBeVisible()
}

The exact values depend on your product. The review question stays simple. Can one neat example hide a serious failure?

5. Read the failure

Run the test against a broken result. Then read its message.

Another engineer should understand the problem without opening the whole file. Compare these messages:

Expected: "$20.00"
Received: "$120.00"

Timeout after 30000ms

The first message points to the product error. The second sends someone searching through logs.

Use clear result checks and useful test names. A failure should shorten the investigation.

6. Run it twice

Use the same input twice. You should get the same result.

Repeated runs catch shared data and timing problems. They also expose tests that depend on another test.

Do not accept a passing second run as proof. Compare both runs. Investigate any difference before keeping the test.

7. Make the release decision

Finish with one question: Would this test stop a bad release?

Name the release it could stop. For example: "This test blocks checkout when totals are wrong."

Keep the test when the answer is clear. Rewrite or delete it when the answer stays vague.

This step protects the review queue. Teams do not need every generated test. They need the small set that proves important behavior.

A five-minute review example

Imagine that AI writes a checkout test. The test adds one product and completes payment. It checks the success message.

First, name the risk. A customer could pay the wrong total.

Next, change the price calculation. The test still passes because the message appears. You found the missing proof.

Add a check for the order total. Then try an expired coupon and an empty cart. The test should explain each failed result.

Run the test twice with the same data. Both runs should match.

Finish with the release question. This test should stop checkout when the charged total is wrong. The answer now names one clear release failure.

That review improved one test without adding more code than needed. The team gained useful proof instead of another passing file.

Use the card during review

You can complete this review in a few minutes:

Name the user risk.
Break the protected behavior.
Check the final result.
Change the input data.
Read the failure message.
Repeat the same run.
Name the bad release it stops.

AI can handle test volume. You still decide what deserves trust. Keep tests that fail correctly and explain why the release should stop.

Test Retries Hide Real Bugs: When a Rerun Helps and When It Lies

Anton Gulin — Wed, 05 Aug 2026 10:00:00 GMT

Post

Your build is green. One of your tests failed five minutes ago. Both of those things are true, because the test passed on a retry. Here is the short answer. A retry never fixes anything. Keep retries on in CI, but treat a retried pass as a bug report. It opens a ticket. It never closes one.

Last Saturday I posted about rerunning failed tests in isolation. Two readers pushed back, and both were more right than my post.

The first said flakiness is a signal. Turning the alarm off does not put out the fire.

The second was more specific. A team reruns the failed test alone. It passes. Everyone moves on. And the real cause, a race condition, ships to production. A race condition means two things run at the same time and collide.

This post is the concession, and the setup I now recommend.

What the rerun actually changed

Look at what happens when a test fails in CI and someone reruns it alone.

The code did not change. The data did not change. One thing changed: the test ran without the other tests around it.

So the rerun did not prove the test is fine. It proved the test is fine when nothing else is running. That is a different sentence. Your users do not visit your app one at a time.

The race that a rerun hides

Here is the simplest version of the trap:

test('admin can rename a user', async () => {
  await renameUser('user-42', 'New Name');
  await expect(profileName).toHaveText('New Name');
});

test('report shows user names', async () => {
  const report = await openReport();
  await expect(report.row('user-42')).toContainText('New Name');
});

Both tests touch the same record. Run together, they sometimes collide, and one fails. Run alone, each passes every time.

The failing run was the only honest one. It was telling you the app has a timing bug. The rerun in isolation deleted the evidence.

Playwright already tells you the truth

Here is the part most teams never look at. When a Playwright test fails and then passes on a retry, the report does not say "passed."

It says flaky.

  3 passed
  1 flaky

The tool is honest. The habit is not. Most dashboards only show green or red, so a flaky pass reads as a pass, and the count nobody reads keeps growing.

The setup that keeps retries useful

Retries have one honest job: telling infrastructure noise apart from real signal. A container that started slowly is noise. A race condition is signal. You cannot tell them apart without evidence, so collect it:

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

Two lines, two jobs. The retry detects. The trace records everything about the failing run, so you can diagnose it later instead of shrugging.

Locally, retries stay at zero. On your own machine you want the failure loud and immediate.

The rule

A retried pass opens a ticket. It never closes one.

Once a week, read the flaky list. Every entry is one of two things. Infrastructure noise you should fix in the pipeline. Or a real timing bug in the app that your tests found first. Both are work. Neither is "passed."

My Saturday post said rerunning a flaky test in isolation makes things worse. These two readers explained the mechanism better than I did. The rerun does not just waste time. It manufactures false confidence and deletes the only evidence you had.

The green build is not the goal. The true build is.

Reuse One Page Object Method for Success and Failure Cases

Anton Gulin — Wed, 29 Jul 2026 10:00:00 GMT

Post

Your login helper exists twice. One version expects the dashboard. One version expects the error. Ninety percent of the two bodies is the same code. Here is the short answer. Keep one method per user action. Let an options object say what it waits for. The test still decides what the result means.

Last week I wrote about where test assertions belong. A reader came back with the harder version of the question. If business checks live in the test, how does one method serve both cases? Good question. This post is the answer.

The duplicate everyone has

Open any page object that is more than a year old. You will find this:

async login(user: string, pass: string) {
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  await this.page.waitForURL('**/dashboard');
}

async loginExpectingError(user: string, pass: string) {
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  await this.errorBanner.waitFor();
}

Two methods. Three identical lines each. One real difference: what the method waits for at the end.

Now the login form changes. Someone adds a "remember me" checkbox. You fix the first method. You forget the second. The negative test starts failing for a reason unrelated to the negative case.

That is the cost of the duplicate. Not ugliness. Drift.

Why the boolean flag makes it worse

The first fix everyone tries is a flag:

async login(user: string, pass: string, success: boolean) { ... }

Then the test reads like this:

await loginPage.login('anton', 'wrong-password', false);

False what? You have to open the page object to find out. And flags multiply. Six months later the same method takes three of them, and nobody can read the call site at all:

await loginPage.login('anton', 'pw', false, true, false);

A flag saves a method and costs a reader. Bad trade.

Copy the shape Playwright already uses

Playwright's own API solves this everywhere. Every action takes an optional options object with named keys:

await page.getByRole('button').click({ timeout: 5000, force: true });

You never pass a bare true to Playwright. You pass a name and a value. Do the same in your page objects:

type LoginOptions = { waitFor?: 'dashboard' | 'error' };

async login(user: string, pass: string, options: LoginOptions = {}) {
  const { waitFor = 'dashboard' } = options;
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  if (waitFor === 'dashboard') {
    await this.page.waitForURL('**/dashboard');
  } else {
    await this.errorBanner.waitFor();
  }
}

Two things make this work. The default keeps the common call short. The named key makes the rare call readable:

// happy path, unchanged
await loginPage.login('anton', 'correct-password');

// negative path, and you can read it without opening the class
await loginPage.login('anton', 'wrong-password', { waitFor: 'error' });

One method. One place to fix when the form changes.

Notice what the option does not say

The option is called waitFor, not expectSuccess. That is deliberate.

expectSuccess: false puts a judgment inside the page object. The class starts deciding what a correct login looks like. That breaks the rule from last week: business checks live in the test.

waitFor: 'error' only says which element the method should wait for before it returns. It is a timing instruction, not a verdict. The verdict stays where a reader expects it:

test('rejects a wrong password', async ({ loginPage }) => {
  await loginPage.login('anton', 'wrong-password', { waitFor: 'error' });
  await expect(loginPage.errorBanner).toHaveText('Wrong password');
});

Read that test aloud. It says what it does and what should be true. You never open the page object.

When two methods really are right

Sometimes the duplicate is not a duplicate. Keep separate methods when the steps themselves differ:

async loginWithPassword(user: string, pass: string) { ... }
async loginWithSso(user: string) { ... }

Different fields, different clicks, different waits. These are two user actions that happen to end in the same place.

The test: if the bodies differ by more than the final wait, keep them apart. If they differ only by the final wait, merge them.

And name methods after the action, never after the outcome. loginExpectingError names a result. login names what the user does. Results belong in the test name.

The rule, in three lines

One method per user action.
An options object says what it waits for, with a default for the common case.
The test says what the result means.

Apply it once to your login helper. Then look at your checkout helper, your search helper, and your upload helper. The same pair is usually hiding in all of them.

Should Page Objects Assert? Where Test Assertions Belong

Anton Gulin — Wed, 22 Jul 2026 10:00:00 GMT

Post

Should a page object contain assertions? Here is the short answer: business assertions belong in the test. Technical guards belong in the page object. A test must show what "correct" means for its scenario. A page object must only promise that the page is ready to use. Mixing the two is why suites become hard to read and hard to trust.

Last week I published the modern page object model. One sign of legacy design was "page objects that assert." A senior QA engineer pushed back in the comments. His position: business checks go in the test layer, but technical checks, did the page reach the right state?, can hide inside the class.

It was a fair challenge. This post is the full answer.

The two kinds of checks

Every check in a UI test is one of two kinds.

Business checks answer: did the product do the right thing? The order total is $41.97. The welcome message names the user. The discount applied.

Technical guards answer: is the page ready? The form finished loading. The spinner went away. The URL changed.

They look similar in code. They serve different readers. A business check speaks to the person deciding "is this feature broken?" A technical guard speaks to the machine deciding "can I click now?"

The rule

Put business checks in the test, always:

test('applies the discount', async ({ checkoutPage }) => {
  await checkoutPage.applyCode('SAVE10');
  await expect(checkoutPage.total).toHaveText('$35.97');
});

The expected value sits in the test file. When this fails at 2 a.m., the reader sees what "correct" was supposed to be. No file jumping.

Handle technical guards inside the page object, but prefer waiting over asserting:

export class CheckoutPage {
  readonly total: Locator;

  async applyCode(code: string) {
    await this.codeInput.fill(code);
    await this.applyButton.click();
    await this.priceUpdate.waitFor({ state: 'visible' });
  }
}

The page object does not judge the total. It makes one promise: when applyCode returns, the page finished reacting. That is a technical guard, and note it is a wait, not an assert. Playwright's web-first assertions and auto-waiting handle most of these guards for free.

Why hidden business asserts hurt

Three costs show up at scale.

The expected value disappears. checkoutPage.verifyTotal() hides $35.97 in another file. The failing test cannot tell you what it believed.

The page object takes sides. Fifty tests share that class. One scenario needs a different expected total, and the shared method becomes a maze of parameters.

Failures point at the wrong layer. When an assert fires inside a page object, the stack trace blames plumbing. The reader has to dig to find which business rule broke.

Where it gets honestly debatable

My commenter's position (hidden technical assertions are fine) is workable. Plenty of strong suites do it. My preference is stricter for one reason: an assert stops the test with a verdict; a wait just holds the door. Verdicts belong to tests. But if your team hides technical guards as asserts and everyone can read the failures, that is a style choice, not a defect.

What is a defect: expected business values living anywhere except the test body.

The migration path

Same as the POM modernization: no big rewrite. When a change touches a page object that asserts, move the business expectation up into the tests that call it, and convert the technical remainder into a wait. Each class takes minutes.

Run this to find your candidates:

grep -rn "expect(" src/pages/

Every hit is either a business check to promote or a guard to convert.

The takeaway

Business checks: in the test, expected values written out.
Technical guards: in the page object, as waits, not verdicts.
Playwright's auto-waiting already covers most guards, delete before you migrate.

The Modern Page Object Model: Less Shared Code, Easier Changes

Anton Gulin — Wed, 15 Jul 2026 10:00:00 GMT

A page object model (POM) organizes browser tests by app page. Each page or component gets one class. The class stores element finders, called locators. It also stores user actions. Tests call those actions instead of using selectors directly. When a page changes, you fix one class. You do not fix two hundred tests.

The basic idea is old. Modern Playwright changes how we should build it.

Many page objects still follow Selenium tutorials from 2015. They work, but they ignore useful Playwright features. Here are five signs and the simpler pattern that replaces them.

Sign 1: Every page object extends `BasePage`

export class LoginPage extends BasePage {
  constructor(driver: Driver) { super(driver); }
}

Inheritance means one class receives behavior from a parent class. The 2015 logic made every page share one parent. That parent often becomes a junk drawer. It collects waits, logs, screenshots, and unused helpers.

Inheritance ties every page to the parent class. Modern page objects need less shared code. Shared helpers can live in plain functions. Test setup can live in fixtures, which prepare objects for each test.

Sign 2: Wait methods everywhere

await loginPage.waitForPageToLoad();
await loginPage.waitForSpinnerToDisappear();

Playwright checks elements before many user actions. For click, it checks visibility, stability, events, and enabled state. Many manual wait methods only repeat those checks. They can also hide timing bugs.

Wait explicitly only for conditions Playwright cannot infer. One example is a dashboard finishing a calculation. Use an assertion, which checks an expected result:

await expect(total).toHaveText('41.97');

Sign 3: Locators buried inside methods

async login(user: string, pass: string) {
  await this.page.locator('#username').fill(user);
  await this.page.locator('#password').fill(pass);
  await this.page.locator('button[type=submit]').click();
}

Hidden locators make page dependencies hard to see. Declare each locator once in the constructor:

export class LoginPage {
  readonly username: Locator;
  readonly password: Locator;
  readonly submit: Locator;

  constructor(readonly page: Page) {
    this.username = page.getByLabel('Username');
    this.password = page.getByLabel('Password');
    this.submit = page.getByRole('button', { name: 'Sign in' });
  }

  async login(user: string, pass: string) {
    await this.username.fill(user);
    await this.password.fill(pass);
    await this.submit.click();
  }
}

Now every locator is visible at a glance. The code also uses user-facing names through getByRole and getByLabel. Those names often survive redesigns. A CSS selector like #username may not.

Sign 4: Tests build their own page objects

test('login works', async ({ page }) => {
  const loginPage = new LoginPage(page);   // every test, every file
  ...
});

This setup repeats in every test. A Playwright fixture prepares the page object once:

// fixtures.ts
import { test as base } from '@playwright/test';
import { LoginPage } from './pages/login-page';

export const test = base.extend<{ loginPage: LoginPage }>({
  loginPage: async ({ page }, use) => {
    await use(new LoginPage(page));
  },
});

// login.spec.ts
import { test } from './fixtures';

test('login works', async ({ loginPage }) => {
  await loginPage.login('anton', 'secret');
});

The test asks for loginPage and receives a ready object. There is no repeated setup. One fixture can also depend on another fixture.

Sign 5: Page objects that assert

A loginPage.verifyDashboardIsCorrect() method lets the page object define "correct." The expected result now hides inside a shared class.

Modern split: page objects act, tests assert. The page object returns locators or values. The test states the expectation in plain sight:

await loginPage.login('anton', 'secret');
await expect(dashboard.greeting).toHaveText('Welcome, Anton');

When this fails, the expected value is visible in the test. You know what broke.

The migration path (no rewrite required)

You do not need a full rewrite:

New page objects follow the modern shape from day one.
When a test touches an old class, upgrade only that class.
Delete extends BasePage and move locators into the constructor.
Delete duplicate wait methods and move assertions into tests.
Add fixtures early. New tests get clean setup while old tests keep working.

Do AI agents change any of this?

They make this structure more important. Page objects give an AI agent a small list of approved actions. Raw selectors make the agent rediscover each page. That creates fragile tests. Calling loginPage.login() uses the same reviewed action as a human-written test.

Small, flat page objects give machines fewer ways to make mistakes. Large parent classes give them more choices.

The pattern in one line

Declare locators in the constructor, user-facing. Let fixtures do the wiring. Let auto-waiting do the waiting. Keep assertions in tests. Share nothing through inheritance.

Your page objects should be the most boring code you own. That is what makes them last another ten years.

Anton Gulin is the AI QA Architect &#8212 the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

Your AI Model Is a Dependency: Pin It, Keep a Fallback, Re-Verify

Anton Gulin — Wed, 08 Jul 2026 10:00:00 GMT

Nineteen days. That is how long one of the world's best AI models was simply gone this summer.

Claude Fable 5 went offline on June 12 by government order. It came back on July 1. Between those dates, every test suite that leaned on it had a problem. Some teams fixed it in one line. Others lost days, twice.

The difference was not luck. It was whether they treated the model like what it is: a dependency.

What "the model is a dependency" actually means

Your test suite already has dependencies. A database version. A browser version. A Node version. You pin them all. (Pinning means locking the exact version so nothing upgrades by itself.)

An AI model is the same kind of moving part, with three extra ways to hurt you:

It changes silently. Providers update models behind the same name. The model you tested in March is not always the model answering in July.
It disappears. Deprecations happen every quarter. And as of this summer, so do government-ordered shutdowns. Fable 5 was not deprecated — it was switched off overnight, worldwide.
It gets swapped without telling you. The fine print of the Fable 5 return says blocked requests are rerouted to a different model (Opus 4.8). Your pipeline asks one brain and sometimes gets another.

A test that stands on a part like this, unpinned and unwatched, is not a test. It is a hope.

Rule 1 — Pin the version

Never let a test suite float on "latest" or on a bare model family name. Point it at the exact model ID, in one place.

// config/models.ts — ONE place the whole suite imports from
export const MODELS = {
  primary: "claude-fable-5",      // pinned: the exact ID we validated
  fallback: "claude-opus-4-8",    // pinned: the exact ID we validated
} as const;

Every agent, every AI-assisted test imports from this file. Nothing names a model directly. When the world changes, you edit one line, not forty files.

If you use an agent framework, the same rule applies to its config. In Stagehand, for example, the model is an explicit setting — set it, do not rely on an implicit choice:

const stagehand = new Stagehand({
  modelName: MODELS.primary,   // never omit this
});

Rule 2 — Keep a fallback you have already validated

A fallback you pick during the outage is not a fallback. It is a gamble made under pressure.

Pick the second model now, on a calm day. The bar is simple: it must run your real suite acceptably. Not "it is a good model" — your suite, green, at a cost you accept.

Write it into the same config file (see MODELS.fallback above). One line to switch, pre-approved, pre-tested.

Rule 3 — Re-run the same suite on both

This is the rule most teams skip, and it is the one that makes the other two real.

A fallback is only valid while it stays valid. Models drift. So, on a schedule, run your same test suite against both models and compare:

// parity.spec.ts — the same checks, both brains
for (const model of [MODELS.primary, MODELS.fallback]) {
  test(`checkout flow passes on ${model}`, async () => {
    const agent = makeAgent(model);
    await agent.run("complete checkout for the seeded cart");
    // The fixed check the agent cannot move — same for both models:
    expect(await getOrderTotal()).toBe(41.97);
  });
}

Two details carry this pattern:

The assert is model-independent. The known answer (41.97 for a seeded cart) comes from your test data, not from the model. Either brain must reach the same truth.
Run it weekly, not "someday." A parity suite that ran last night makes a model switch a non-event. A parity suite that never runs makes it an emergency.

What this cost the two kinds of teams

When Fable 5 went dark, the pinned teams with a validated fallback did this: change one line, run the suite, ship. When it came back July 1, they did it again in reverse. Two boring mornings.

The floating teams debugged broken builds twice in three weeks — once out, once back. Same event, same tools. The whole difference was written before the crisis: one config file, one spare model, one recurring suite.

The checklist

Every model reference goes through one config file
Model IDs are exact and pinned — no "latest", no bare family names
A fallback model is chosen, and it has passed your real suite
A parity suite runs both models on a schedule
A model switch is documented as a one-line change anyone can make

You would not float your database version in production. Your AI model earns the same respect — it is infrastructure now, and this summer proved it can vanish like infrastructure too.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

How to Implement AI in QA (2026): A Practical Framework

Anton Gulin — Wed, 01 Jul 2026 10:00:00 GMT

How to Implement AI in QA (2026): A Practical Framework

Key takeaways

To implement AI in QA, let AI do the work and let a fixed check decide the result. Use AI to generate tests, repair broken selectors, and explore your app. Never let AI grade its own output. Add an independent check the AI cannot change, test for repeatable results, and test that the agent did only what you asked.

What "AI in QA" actually means

AI in QA means using an AI model to help test software. That is the whole idea. The model can write test cases, fix tests that broke, click through your app like a user, or read a failure and guess the cause.

It does not mean the AI replaces testing. It means the AI does some of the work a tester used to do by hand. The judgment stays with you.

I test software for a living. The teams that win with AI in QA all draw the same line. AI does the work. A human, and a fixed check, decide if the work is good.

The one rule that keeps it safe

Here is the rule the rest of this guide hangs on.

Let AI do the work. Never let AI judge its own work.

Picture an AI agent that tests your login. It clicks around. It reports green. Everyone relaxes. But the agent decided what "pass" means. If it is too kind, it passes a broken page. Now you shipped a bug with a green check on top.

An AI that grades its own work is not a test. It is an opinion.

So you give the AI room to explore, and you keep one thing it cannot touch. A fixed check. A known-good answer. A hard assert (a check that fails loudly) on the real outcome. The agent finds the path. The fixed check says pass or fail.

In testing this fixed answer has a name: an oracle. The oracle is the part the system being tested is not allowed to influence. Keep your oracle out of the AI's reach and most AI-in-QA risk goes away.

Where AI helps in QA (the 4 good jobs)

These four jobs are where AI pays off today.

Writing tests. Point the model at a page or a user story. It drafts test cases, including edge cases a tired human skips. You review and keep the good ones.
Fixing broken tests. A button moved and the test broke. AI can find the new selector (how a test finds a button) and propose the fix. This is the biggest time-saver for most teams.
Exploring the app. An AI agent can wander your app like a curious user and report what feels broken. Great for finding the bug nobody wrote a test for.
Reading failures. When a test fails, AI can read the log and the trace and suggest the likely cause. It turns a wall of red into a short list to check.

In all four, the AI proposes. You and your fixed checks dispose.

Where AI lies in QA (the 4 traps)

This is the part most guides skip. AI in QA fails in four ways. Plan for each.

It passes a broken thing. A too-kind agent calls a broken page "fine." Fix: an independent oracle the agent cannot move.
It is not repeatable. The same input passes now and fails in ten minutes. Fix: run the same input twice and compare the shape of the answer.
It does too much. You asked for one thing. The agent also changed a setting or sent a message. Fix: a scope check on what it touched.
It depends on a model that can change. The model under your tool updates, or even goes offline, and your tests shift with it. Fix: pin the model version and keep a fallback.

That last one is not theoretical. In June 2026 a widely used model was pulled offline overnight. Teams that pinned their model switched to a fallback in one line. Teams that did not found out when their build broke.

A working example: an AI agent with an oracle it cannot move

Here is the pattern in code. An AI agent books a meeting room. Then three fixed checks decide if it really worked. The agent never grades itself.

import { test, expect } from '@playwright/test';
import { Stagehand } from '@browserbasehq/stagehand';

test('AI books a room — and only that', async ({ page }) => {
  // Pin the model. Do not let it auto-upgrade under your tests.
  const stage = new Stagehand({ env: 'LOCAL', model: 'anthropic/claude-opus-4-8' });
  await stage.init();

  // 1) Let the AI do the work. It decides HOW to book the room.
  await stage.act('Book room B for 2pm tomorrow, for 30 minutes');

  // 2) The fixed check the AI cannot move (the oracle).
  //    These helpers read your real database, not the agent's report.
  const booking = await getBookingFromDb({ room: 'B', time: '14:00' });
  expect(booking).toBeTruthy();          // it did the task
  expect(booking.durationMin).toBe(30);  // exactly what we asked for

  // 3) Scope check. Did it touch anything it should not have?
  const otherChanges = await getChangesExcept(booking.id);
  expect(otherChanges).toHaveLength(0);  // no surprise side effects
});

Read the three checks again. The agent's own "I booked it" is never trusted. The database is the oracle. The duration check catches a sloppy booking. The scope check catches the agent doing extra. (getBookingFromDb and getChangesExcept are your own helpers — they read real state, not the agent's words.)

To catch the repeatable-result trap, run the same prompt twice and compare:

test('same request, same result', async () => {
  const a = await runBooking('Book room B for 2pm tomorrow, 30 minutes');
  const b = await runBooking('Book room B for 2pm tomorrow, 30 minutes');
  // The wording of the agent's reply may differ. The outcome may not.
  expect(a.room).toBe(b.room);
  expect(a.durationMin).toBe(b.durationMin);
});

The model is allowed to phrase its answer differently each time. It is not allowed to book a different room.

The 3 tests every AI feature needs

If you ship an AI feature to users, these three tests catch the failures that page you at 2am. Most teams only write the first easy one ("does it give a good answer?").

The wrong-input test. Feed it junk, empty fields, another language, a user trying to break it. A good feature fails safely. A bad one is confidently wrong.
The same-input-twice test. Run the exact input twice. Same kind of answer? Different wording is fine. Pass-then-fail is not.
The scope test. Did it do only what you asked? Or did it also change a setting, send a message, or touch a file? Extra is not helpful. Extra is a future incident.

Where to start on Monday

You do not need a platform or a budget. Start small.

Pick one flaky test. Let an AI tool propose the fix. You keep final say.
Add one oracle. Take your most important flow and add a hard check on the real outcome, not the agent's report.
Pin your model. Lock the version your AI tools use. Add a fallback.
Add the scope check to one agent run. See what it touches when you are not looking.

Do those four and you have AI in QA that you can trust. The AI does more work. You keep the judgment. The fixed checks keep everyone honest.

That is the whole job: the gap between "the AI says it passed" and "it passed, for the right reason."

How to Test Passkey (WebAuthn) Login in Playwright (2026)

Anton Gulin — Fri, 26 Jun 2026 03:24:00 GMT

How to Test Passkey (WebAuthn) Login in Playwright (2026)

Key takeaways

You can now test passkey login in Playwright with no hardware key. Playwright 1.61 added a virtual authenticator (a fake security key). Your test seeds a passkey, turns it on, and the page signs in as if a real key answered. It works in every browser and runs in CI. The API is browserContext.credentials, with three methods: create(), install(), and get().

What is a passkey, in plain words

A passkey is a login with no password. You sign in with your face, your fingerprint, or a device PIN. The hard part lives in your device. The website only sees a signed reply.

The browser standard behind this is called WebAuthn. It means "web authentication". When you log in, the browser runs a small back-and-forth with the site. The site asks. Your device answers and signs. This is the part that used to need real hardware.

Apple, Google, and most banks ship passkeys now. If your app has a "sign in with a passkey" button, you have this flow in production today.

Why nobody tested this flow

Here is the part nobody talks about. Almost nobody tests the passkey login.

For years it was hard. To test a passkey you needed a real security key plugged into the machine. You cannot plug a USB key into a CI server. CI is a remote build machine with no hands and no ports.

So the most important login flow shipped untested. The one thing a user does first. The one thing that locks them out if it breaks.

I test software for a living. An untested login is the scariest gap on the list. If sign-in breaks, nothing else matters. The cart, the dashboard, the settings page, all of it sits behind the door. On one project I watched a broken auth path block every other test for two days. The fix took ten minutes. Finding it took the two days.

Playwright 1.61 closed this gap.

What Playwright 1.61 added

Playwright 1.61 shipped on June 15, 2026. It added a virtual authenticator (a fake security key).

A virtual authenticator is software that pretends to be a hardware key. Your test creates one. It seeds a passkey into it. From then on, when the page calls the browser to sign in, Playwright answers for the key. No real device. No USB port. The page cannot tell the difference.

You reach it through a new class called Credentials, on the browser context: browserContext.credentials. It has three methods you will use most:

create() seeds a test passkey for a site.
install() turns the virtual key on for the page.
get() reads back any passkey the page registered, so you can save it and reuse it later.

This works in all three browser engines Playwright drives. So one test covers Chromium, Firefox, and WebKit.

Here is a complete test. It seeds a passkey, turns on the virtual key, then signs in. Read the comments for what each step does.

// passkey-login.spec.ts
// Tested against Playwright 1.61. Run with: npx playwright test
import { test, expect } from '@playwright/test';

test('user signs in with a passkey', async ({ browser }) => {
  // A fresh, clean browser session for this test.
  const context = await browser.newContext();

  // STEP 1 — Seed a passkey for our site.
  // 'example.com' is the site domain (the "relying party id").
  // With only the domain, Playwright makes a fresh key for us.
  await context.credentials.create('example.com');

  // STEP 2 — Turn the virtual key on.
  // From now on, the page's sign-in calls are answered by our key,
  // not by real hardware. Call this before the page loads.
  await context.credentials.install();

  // STEP 3 — Let the page use it.
  const page = await context.newPage();
  await page.goto('https://example.com/login');

  // The page calls the browser to sign in. Our key answers.
  await page.getByRole('button', { name: 'Sign in with a passkey' }).click();

  // Check the user is in.
  await expect(page.getByText('Welcome back')).toBeVisible();
});

The three steps map to the three method calls. First you create() a passkey for your site. Then you install() the virtual key, which makes the page's sign-in calls run through it. Then the page does its normal login, and your key answers in place of hardware.

One note on order. Call install() before the page touches sign-in. The virtual key only answers calls that happen after you turn it on.

Re-using a passkey across tests

Often you want to register a passkey once, then reuse it in many tests. A passkey holds a private key (a secret only your device knows). You can read that secret back with get() and seed it into a later test.

// In a setup test: register once, then read the passkey back.
const created = await context.credentials.get({ rpId: 'example.com' });
// `created` holds the passkey fields, including its keys.
// Save them, then seed an identical passkey in a later test:
await otherContext.credentials.create('example.com', {
  id: created[0].id,
  userHandle: created[0].userHandle,
  privateKey: created[0].privateKey,
  publicKey: created[0].publicKey,
});

This is how you keep one stable test user across a whole suite. You do not re-register on every test. You seed the same passkey each run. See the Credentials docs for the full field list.

The honest part: what this does not test

A virtual key is not a real key. So this approach tests your login flow, not the physical hardware. It will not catch a bug in a specific phone's secure chip or a real fingerprint reader. It tests the part you own: the page, the back-and-forth, the server check.

For most teams that is the right line. The browser and the operating system test the hardware path for you. Your job is to test that your app asks the right question and trusts the right answer. That is exactly what the virtual key lets you do, in CI, on every push.

Why this matters for your CI pipeline

Before 1.61, your passkey login had two test options. Skip it, or test it by hand. Both are bad. A skipped test means a silent break. A by-hand test runs once a release, not once a push.

Now it runs like any other test. It sits in your suite. It runs on every pull request. If someone changes the login and breaks the passkey path, the build goes red before the change ships. That is the whole point of a test. Catch the break in seconds, not from an angry user.

If your app supports passkeys, this is the test you write this week. The excuse is gone. The login everyone ships and nobody verifies is now testable.

Related reads on anton.qa:

Official sources:

Playwright Best Practices: 10 Rules AI Agents Get Wrong (2026)

Anton Gulin — Wed, 17 Jun 2026 10:00:00 GMT

Playwright Best Practices: 10 Rules AI Agents Get Wrong (2026)

Playwright best practices are the rules that keep browser tests stable and easy to read. Use role-based locators (find by what users see), web-first assertions that auto-wait, and isolated tests. Seed data through the API (direct requests), not the UI. Avoid hard waits, conditional logic, and tests tied to your HTML. Turn on traces and run in parallel.

An AI agent can write 50 Playwright tests in a minute. That feels fast.

Then those tests fail at random, and nobody knows why. The agent copied old patterns from its training data. It does not know the run failed last night.

This guide lists the 10 best practices that keep tests stable. For each one, I show a small correct example. I also show what AI code agents get wrong. AI tools like Copilot, Cursor, and even Playwright codegen (the test recorder) lean on stale habits. Someone has to fix that.

1. Find elements the way a user sees them

A locator (a pointer to an element) should match what a person sees on screen. Use getByRole, getByLabel, or getByText. These read like the page. They also survive a redesign of your HTML.

import { test, expect } from '@playwright/test';

test('user can sign in', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('ada@example.com');
  await page.getByRole('button', { name: 'Sign in' }).click();
});

What AI agents get wrong: they reach for CSS or XPath (brittle path selectors) like page.locator('div.btn-primary > span'). Change one class name and the test breaks.

2. Use web-first assertions that wait for you

A web-first assertion (a check that auto-waits) retries until the page is ready. expect(locator).toBeVisible() waits on its own. You never add a fixed sleep.

import { test, expect } from '@playwright/test';

test('welcome message appears', async ({ page }) => {
  await page.goto('/dashboard');
  await expect(page.getByText('Welcome back')).toBeVisible();
});

What AI agents get wrong: they add await page.waitForTimeout(3000) (a hard pause). Hard waits are the top cause of flaky tests (tests that fail at random). Too short, the test fails. Too long, the suite crawls.

3. Keep every test isolated

Isolated means each test starts clean. No shared login. No leftover data from the test before. Playwright gives each test a fresh browser context (a clean session). Set up state in a hook, not across tests.

import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('ada@example.com');
  await page.getByRole('button', { name: 'Sign in' }).click();
});

test('shows the account name', async ({ page }) => {
  await expect(page.getByRole('heading', { name: 'Ada Lovelace' })).toBeVisible();
});

What AI agents get wrong: they chain tests, where test 2 needs test 1 to run first. One failure then breaks the whole file.

4. Seed state through the API, not the UI

To test a page, you often need data first. A user, an order, a draft. Do not click through ten screens to make it. Send the data straight to your backend with the request fixture (a built-in HTTP client). It is faster and steadier.

import { test, expect } from '@playwright/test';

test('opens an existing project', async ({ page, request }) => {
  const res = await request.post('/api/projects', {
    data: { name: 'Apollo' },
  });
  expect(res.ok()).toBeTruthy();

  await page.goto('/projects');
  await expect(page.getByText('Apollo')).toBeVisible();
});

What AI agents get wrong: they build the data through the UI every time. The test gets long and slow, and a setup step fails for reasons that have nothing to do with the real check.

5. Do not lean on test IDs by default

A test ID (a tag added just for tests, like data-testid) works as a fallback. But reach for getByRole and getByLabel first. Those test what a real user can do. A test ID only proves an attribute exists.

import { test, expect } from '@playwright/test';

test('cart shows one item', async ({ page }) => {
  await page.goto('/cart');
  // Prefer a real role over a test id.
  await expect(page.getByRole('listitem')).toHaveCount(1);
});

What AI agents get wrong: they paste data-testid on everything. The tests pass even when the button has no label and a screen reader (assistive software) cannot find it. The test misses a real bug.

6. Turn on traces for the first retry

A trace (a full recording of the run) shows every step, the DOM, and the network. Set it to record only on the first retry of a failed test. You get the evidence for failures, and clean runs stay fast.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 1,
  use: {
    trace: 'on-first-retry',
  },
});

What AI agents get wrong: they leave tracing off, or set trace: 'on' for every run. Off means no evidence when a test fails. Always-on slows the suite and fills your disk.

7. Run tests in parallel and shard them

Parallel means many tests run at once. Playwright does this by default. For one big file of independent tests, set parallel mode. To split a slow suite across machines, use sharding (run a slice per machine).

import { test, expect } from '@playwright/test';

test.describe.configure({ mode: 'parallel' });

test('loads home', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/Home/);
});

Split across three machines on CI (your build server):

npx playwright test --shard=1/3

What AI agents get wrong: they write tests that share a database row or a single user. Run those in parallel and they fight each other, so you get flaky tests.

8. Keep `if` and `try` out of your tests

A test should walk one clear path. No branching. If a test asks "is the button there? if so click it," it hides a bug. The button should always be there. Assert it.

import { test, expect } from '@playwright/test';

test('checkout button works', async ({ page }) => {
  await page.goto('/cart');
  // Assert the state. Do not guess it with an if.
  const checkout = page.getByRole('button', { name: 'Checkout' });
  await expect(checkout).toBeEnabled();
  await checkout.click();
});

What AI agents get wrong: they wrap clicks in if (await locator.isVisible()) to stop errors. That hides the real failure. A test that skips its own check still goes green.

9. Test what users see, not how it is built

Test the behavior, not the internals. Check the visible result. Do not check a CSS class, a state variable, or a function name. Those change when you refactor (rewrite the code), even though the app still works.

import { test, expect } from '@playwright/test';

test('shows a success message after submit', async ({ page }) => {
  await page.goto('/contact');
  await page.getByLabel('Message').fill('Hello');
  await page.getByRole('button', { name: 'Send' }).click();
  // Check the user-facing result, not an internal class.
  await expect(page.getByText('Thanks, we got your message')).toBeVisible();
});

What AI agents get wrong: they assert on class="is-active" or an exact HTML shape. The test breaks on every redesign, even when nothing real changed.

10. Define projects in your config

A project (a named test setup) in playwright.config.ts runs the same tests under different settings. Use projects to cover Chromium, Firefox, and WebKit (the three main browser engines). One config, full coverage.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  retries: 1,
  use: { trace: 'on-first-retry' },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
});

What AI agents get wrong: they hard-code one browser, or copy a config with no projects array. The suite then tests Chrome only, and a Safari-only bug ships to users.

The human still owns the standard

AI writes the first draft fast. That part is real, and it is useful. But the first draft copies patterns from old code on the internet. It adds hard waits. It clicks through the UI to seed data. It wraps fragile steps in if blocks so the run stays green.

A green suite that proves nothing is worse than no suite. It buys false trust.

So the workflow is simple. Let the agent write the draft. Then a human reads it against these 10 rules and fixes what the agent got wrong. The agent moves fast. The human keeps the tests honest. That is the job of an AI QA Architect.

Build the tests with AI. Then make them stable yourself.

I measure how fast 42 LLMs actually answer. Here's the honest method.

Anton Gulin — Tue, 16 Jun 2026 04:49:00 GMT

I measure how fast 42 LLMs actually answer. Here's the honest method.

I test software for a living. So when a vendor calls an AI model "fast," I don't trust the word. I measure it.

Most leaderboards rank how smart a model is. Almost none rank how fast it answers. You pick a model because it scored well, ship it, and then your users sit and wait.

Speed is two different numbers. People mix them up constantly.

The two numbers

Time to first token (TTFT). The wait before the first word appears. You feel this every time a chatbot "thinks" before replying.

Tokens per second (TPS). How fast the model writes once it starts. A token is a chunk of a word.

A model can be great at one and terrible at the other. You need both.

How I measure it

I run an independent tracker called ollamatps.com. It benchmarks 42 Ollama Cloud models. Here is the exact method, because a benchmark you cannot inspect is just a claim.

One fixed prompt. Every run asks the model to write a 400-word explanation of HTTP routing. Same prompt, every model, every time.
Fixed output cap. max_tokens is capped at 300. It never changes between runs.
Every ~10 minutes. Each model is re-tested continuously, not once.
TTFT is measured from the moment the request is sent to the first non-empty content chunk. It includes network round-trip and prompt-processing time. That is honest, because it is what you actually wait.
TPS is generation throughput only: eval_count / (total_duration - time_to_first_token). The startup wait is removed, so TPS measures pure writing speed.

Same prompt, same cap, same schedule. That is what makes two models comparable.

Building this was a testing job, not a coding job. Retries on failure. A reliability score per model. A circuit breaker for models that keep failing. If you cannot trust the measurement, the number is noise. That part is the same work I do on any test system.

What surprised me

Bigger is not faster.

The fastest model on the board is one of the smallest: a 30B model at over 200 tokens per second. The model literally named "ultra" is dead last, under 8 tokens per second.

And the wait varies wildly. TTFT ranges from about 0.3 seconds to 23 seconds across the 42 models. Same cloud. Roughly 80x difference in how long you wait for the first word.

If you picked your model on a benchmark score alone, you have no idea which of these you are getting.

Why I rebuilt it

The first version tracked fewer models and was less robust. I rebuilt the engine this month (v2) to be multi-provider and to test continuously. The live board updates every 10 minutes.

Watch it run: ollamatps.com

How to Score Your AI Test Agents: Offline Evaluation with Trajectories (2026)

Anton Gulin — Wed, 10 Jun 2026 13:00:00 GMT

How to Score Your AI Test Agents: Offline Evaluation with Trajectories (2026)

AI test agent evaluation is the practice of scoring the tests an AI agent writes, instead of trusting that they pass. You record the agent's run as a trajectory (a saved log of every step), replay it offline, and grade each step for correctness and relevance. Offline scoring needs no live API calls, so you can check agent quality on every pull request.

An AI agent can write 200 tests before lunch. That feels like progress.

Then a real bug ships, and not one of those tests caught it. The agent was confident, and it was wrong.

This guide shows how to stop guessing and start scoring. Stagehand 3.5.0 made the method first-class on June 3, 2026. The pattern works for any agent.

1. It passed is not a score

A green test suite tells you the tests ran. It does not tell you the tests were right.

An AI agent makes three mistakes a human reviewer would catch:

It checks the wrong thing. The test passes, but it never asserts the real behavior.
It writes flaky tests (tests that fail at random). They go green often enough to look fine.
It tests a happy path and skips the edge case that actually breaks.

You cannot fix what you cannot measure. So the first job is a number, not a vibe.

2. Record the run as a trajectory

A trajectory is a saved recording of an agent's run. It captures each step: what the agent saw, what it decided, and what code it produced.

You capture it once, during the agent's normal run.

// Illustrative pattern — confirm the exact Stagehand 3.5 API before use.
const trajectory = await agent.run(task, { record: true });
await saveTrajectory(trajectory, "runs/checkout-flow.json");

The recording is the receipt. Now you can study the run after it finishes, as many times as you want.

3. Replay it offline

Offline means you grade the saved run without calling the live model again. No new API cost. No flaky network. Same input every time.

This matters for two reasons. It makes scoring cheap, so you can run it on every pull request. It makes scoring repeatable, so two engineers get the same result.

// Replay the saved run and score it, with no live API calls.
const run = await loadTrajectory("runs/checkout-flow.json");
const score = await evaluate(run, rubric);

4. Score each step with evaluation types

A single pass/fail hides too much. Grade the run on a few clear axes instead.

Correctness: did the test assert the behavior the task asked for?
Relevance: does each step move toward the goal, or wander?
Stability: would this test pass on a clean re-run, or is it flaky?
Coverage: did the agent test the edge case, or only the happy path?

Stagehand 3.5.0 added evaluation types for exactly this kind of offline scoring. You define the rubric once and apply it to every saved run.

const rubric = {
  correctness: (run) => run.asserts.some(a => a.target === task.goal),
  relevance:   (run) => run.steps.every(s => s.onTask),
  stability:   (run) => run.reruns.every(r => r.passed),
};

A run that scores correctness 7/10, relevance pass, flaky tests 0 is a run you can talk about. "It passed" is not.

5. Wire the score into CI

A score you read once and forget changes nothing. Turn it into a gate.

# CI step: fail the build if the agent's tests score too low.
- run: npx evaluate runs/ --min-correctness 0.8 --max-flaky 0

Now the agent earns trust the same way a junior engineer does. It ships work, the work gets graded, and only graded work reaches production.

6. Where this sits: the Evidence Layer

I design AI test systems on a 3-Layer System:

Orchestration: decides what to test.
Execution: runs the tests, where the agent writes code.
Evidence: proves the work is right.

Most teams build the first two layers and stop. They let the agent write tests and assume the green check means quality.

Offline evaluation is the Evidence Layer. It is the difference between an agent you hope works and an agent you can prove works.

The 5-line checklist

Record every agent run as a trajectory.
Replay it offline, with no live API calls.
Score it on correctness, relevance, stability, and coverage.
Gate your build on the score.
Keep the trajectory, so you can re-grade when the rubric improves.

Build the agent. Then prove it works.

Playwright Codegen: The Complete Guide (2026)

Anton Gulin — Wed, 03 Jun 2026 22:39:38 GMT

Playwright Codegen: The Complete Guide (2026)

Playwright Codegen is a native CLI (command-line interface) tool that generates test scripts automatically as you interact with a browser . It records your actions—like clicks, form inputs, and page navigation—and translates them into clean TypeScript or JavaScript test code.

For most developers, writing test locators (how tests find buttons) takes up 60% of test writing time.

Codegen reduces that time to zero.

Here is how to use it, and how to scale it from a simple draft tool to a full production architecture.

1. How to Launch Playwright Codegen

To start the generator, run this command in your terminal:

npx playwright codegen demo.playwright.dev/todomvc

This launch command opens two windows:

A browser window: This is where you click, type, and record your test steps.
The Playwright Inspector: This is a tool window that displays the generated code in real time.

As you click on the page, the tool writes the test code automatically.

2. Capturing Assertions

A test without assertions (checks to verify behavior) is just a script.

Codegen allows you to record checks directly from the UI.

In the browser window, hover over any element and click one of the check buttons in the toolbar:

Assert Visibility: Verifies if an element is visible on the screen.
Assert Text: Verifies if an element contains specific text.
Assert Value: Verifies the input value of a form field.

This generates standard assertions like await expect(locator).toBeVisible() instantly.

3. Playwright Codegen Best Practices

Generated code is a draft.

To make it production-ready, apply these three rules:

Avoid Hardcoded Wait Times

Codegen does not generate sleep statements.

Playwright uses auto-waiting (waiting for elements to be ready).

Keep it that way.

Do not add manual timeouts.

Use Semantic Locators

Playwright prefers locators that represent user actions.

Codegen generates these by default:

// Good: accessible locator
await page.getByRole('button', { name: 'Submit' }).click();

// Bad: fragile CSS selector
await page.locator('#submit-btn-2').click();

Keep the accessible versions.

They prevent flaky tests (tests that fail randomly).

Isolate Your Auth State

Do not record login steps in every single test.

Use Codegen to save your authentication state once.

Run Codegen with this save option:

npx playwright codegen --save-storage=auth.json

Then, configure your tests to load auth.json before running.

This saves hours of run time in CI (continuous integration servers).

4. The Architectural View: From Draft to System

As an AI QA Architect, I view Codegen as a helper.

It is the entry point of the Execution Layer in the 3-Layer System:

Orchestration: Decides when to run tests.
Execution: The code that runs (where Codegen helps).
Evidence: Gathers logs and traces.

Codegen writes the initial code.

But it cannot design the framework.

It cannot handle API mocks (fake servers).

It cannot govern agentic testing systems (where AI agents write and heal tests).

Use Codegen to build the first block.

Then build the architecture around it.

What Is an AI QA Architect? Role, System, and Skills

Anton Gulin — Wed, 27 May 2026 22:48:00 GMT

What Is an AI QA Architect?

An AI QA Architect designs the test system AI agents run on. They connect requirements, test generation, browser execution, evidence, and human review. The job keeps QA engineers in control. It gives agents a safe place to create, run, fix, and explain tests.

QA means quality assurance.

AI QA Architect is a system role.
It is not a tool title.

The person owns the way AI touches quality work.
That includes the rules, the evidence, and the release gate.

The Simple Definition

An AI QA Architect is responsible for the test architecture behind AI-assisted quality work.

Architecture means the system design.
It answers five questions:

What can the agent read?
What can the agent change?
What evidence must it produce?
Who reviews the result?
What blocks the release?

The role matters because AI drafts tests fast.
Fast tests are not enough.

You need a system that catches weak tests before they land.
You also need evidence humans can review.

Why The Role Exists Now

Most QA systems were built for humans first.

A person read the ticket.
A person chose the test.
A person checked the failure.
A person decided if the release was safe.

AI changes the speed of that loop.

An agent can draft a test from a ticket.
It can inspect a failing trace.
It can suggest a fix.
It can explain a failure.

That power creates a new risk.

If the system has weak gates, AI makes weak work faster.
If the system has strong gates, AI can help the team move safely.

That is the AI QA Architect's job.

The Five Parts Of The System

1. Requirement Intake

The agent needs clean input.

Input can be a user story, bug report, ticket, or spec.
The architect defines what good input means.

A good input names:

user goal
expected result
risky path
test data
release impact

Without this, the agent guesses.
Guessing creates noisy tests.

2. Test Generation

AI can draft tests.
Humans still review them.

A good generation flow has rules.

It tells the agent:

which tool to use
which patterns to follow
what code style to match
what not to test
when to stop

This keeps generated tests inside the team's standards.

3. Execution

Execution means running the test.

For browser checks, tools like Playwright matter.
They produce traces, screenshots, reports, and browser state.

Those artifacts become receipts.
Receipts are proof a human can review.

AI agents need receipts too.
Without receipts, the agent only has text.

4. Failure Analysis

A failed test is not enough.

The system must answer:

Did the app break?
Did the test break?
Did the data change?
Did the environment fail?
Should the release stop?

AI can help sort the evidence.
The human still owns the decision.

5. Release Gate

A release gate decides what ships.

CI means server test runs.
The gate uses CI results, test reports, and risk rules.

An AI QA Architect defines the gate.
They decide which failures block release.
They decide which failures need review.
They decide what evidence is required.

How This Differs From QA Automation

QA automation asks, "Can we run the test?"

AI QA architecture asks, "Can we trust the system around the test?"

That includes automation.
It also includes agents, prompts, data, evidence, review, and release rules.

A QA automation engineer may build the runner.
An AI QA Architect owns the full loop.

How MCP Fits

MCP means Model Context Protocol.
It is a standard way for AI tools to connect with external tools.

MCP testing has two jobs.

First, test the MCP server you build.
That means tools, resources, prompts, and errors.

Second, test software through MCP-enabled tools.
That means the agent can call a browser, database, or API tool.

Both jobs need a test system.

The official MCP Inspector helps developers test MCP servers.
It gives a visual way to inspect capabilities and debug requests.

That is useful.
It is not the full QA system.

The architect still defines the rules around safe use.

Skills An AI QA Architect Needs

You need test automation depth.
You need enough AI knowledge to set safe boundaries.
You need architecture judgment.

Core skills:

Playwright or another browser test tool
CI pipeline design
test data strategy
trace and report review
prompt rules for agents
MCP server testing
release gate design
risk-based testing

LLM means language model.
You do not need to train one.

You do need to know how it fails.
It can hallucinate.
It can skip edge cases.
It can accept weak evidence.

Your system must catch those failures.

What The Role Is Not

An AI QA Architect is not a person who buys AI tools.

It is not a person who tells testers to stop writing tests.

It is not a person who trusts every generated test.

The role is more practical.

You design the system so agents can help safely.
You protect the human decision.
You make quality visible before release.

A Practical Checklist

Use this before adding AI to a test suite.

Can the agent read the requirement clearly?
Can it follow your test patterns?
Can it run the right tool?
Can it produce reviewable evidence?
Can a human approve or reject the result?
Can CI block unsafe releases?
Can the team trace why a test changed?

If the answer is no, fix the system first.

FAQ

What does an AI QA Architect do?

An AI QA Architect designs the test system AI agents use. They define how agents read requirements, generate tests, run checks, and produce evidence. The goal is safer test automation with human review.

Is an AI QA Architect replacing QA engineers?

No. The role protects human review. AI agents can draft tests and inspect failures. Humans still own the quality decision. The architect designs the rules around that work.

What is AI test automation architecture?

AI test automation architecture connects human intent, AI assistance, test execution, evidence, and release gates. It includes prompts, tools, CI, reports, data, and review rules.

How does MCP testing relate to QA?

MCP testing covers two jobs. You test MCP servers you build, and you test software through MCP-enabled tools. Both need clear inputs, safe tool access, and evidence a human can review.

What should a team build first?

Build the evidence loop first. Every test failure should show action, page state, assertion, logs, and release impact. Then add AI help inside that loop.

Start With Evidence

AI QA Architect is a system owner.

The job is not to make AI look smart.
The job is to make quality safer when AI joins the loop.

Start with evidence.
Add agents after that.

Author Bio

Playwright vs Cypress vs Selenium in 2026: The QA Architect Decision Guide

Anton Gulin — Thu, 21 May 2026 02:13:00 GMT

Playwright is the best default for new browser test automation in 2026. It gives cross-browser runs, parallel CI, API checks, and AI-agent evidence in one tool. Cypress still fits JavaScript-heavy teams that want fast local feedback. Selenium still fits legacy grids and strict browser labs.

That is the short answer.

The better answer depends on your system.

If AI agents will read your failures, the question changes.
You are no longer picking only a test runner.
You are picking the evidence layer.

What Changed In 2026

Most comparison posts still ask old questions.

They ask which tool has cleaner syntax.
They ask which tool is easier to learn.
They ask which tool starts faster.

Those questions still matter.
They are no longer enough.

AI agents need proof they can inspect.
Proof means screenshots, traces, browser state, and readable failures.

The human reviewer still owns the decision.
The agent only helps when the evidence is clear.

That is why Playwright now has the default seat.

Pick Playwright When Evidence Matters

Pick Playwright for new end-to-end test systems.
End-to-end means browser checks.

Playwright gives you one model across Chromium, Firefox, and WebKit.
Those are browser engines.
They are how pages run.

That matters for real product risk.

It also matters for AI-agent workflows.
AI agents means tools that act.

Playwright now documents Test Agents.
Those agents plan, generate, and repair tests.

The tool also has strong receipts.
Traces show what happened.
Screenshots show where it happened.
Reports help humans review the failure.

Use Playwright when you need:

browser coverage across engines
parallel CI at scale
trace-based debugging
API and UI checks together
AI-agent review paths
long-term framework ownership

CI means server test runs.

Pick Cypress When Local Feedback Matters Most

Cypress is still useful.

That sentence matters.
Tool debates get lazy when one side becomes a villain.

Cypress can be a strong fit for frontend teams.
It works well when developers want quick local feedback.
It also fits teams already built around Cypress Cloud.

Cypress documents cross-browser testing.
It also documents parallel runs through Cypress Cloud.

That can be enough for many product teams.

Use Cypress when:

your app is JavaScript-first
developers own most browser checks
fast local debugging is the main goal
Cypress Cloud is already approved
browser coverage needs are narrow
the suite is not agent-driven yet

The risk appears later.

As the suite grows, evidence gets more important.
That is where Playwright usually wins.

Keep Selenium When Migration Risk Is Higher

Selenium is not dead.

It is still the right answer for some teams.

Keep Selenium when a grid already exists.
Keep it when policy requires it.
Keep it when migration risk is higher than tool value.

But do not choose Selenium by default for new AI QA work.

You will spend too much time rebuilding the evidence layer.
You will also carry older suite habits forward.

Selenium can be stable.
The question is whether it helps the next system.

The Decision Table

Need	Best default	Why
New AI-agent test system	Playwright	Best evidence path
Broad browser engine coverage	Playwright	One model across major engines
Fast frontend feedback	Cypress	Strong local developer loop
Existing Cypress investment	Cypress	Migration may not pay yet
Legacy grid policy	Selenium	Use what the organization can run
Greenfield QA architecture	Playwright	Better long-term receipts

My Four-Question Test

I use four questions before I choose.

Who reads the failure first?
What proof do they need?
Where will the suite run?
What happens when the UI changes?

If the answer includes AI agents, I lean Playwright.

If the answer is one frontend team, Cypress can fit.

If the answer is legacy policy, keep Selenium.

The Practical Recommendation

Start new projects with Playwright.

Keep Cypress when it already serves the team.

Keep Selenium when migration would create more risk.

Then build the same rule across all three:

Every failed test needs a receipt.

That receipt should show:

what action ran
what page state existed
what assertion failed
what changed before failure
what a human must decide

The tool only matters because the evidence matters.

In 2026, that is the real comparison.

Author Bio

Playwright v1.60 Turns Test Failures Into Evidence

Anton Gulin — Thu, 14 May 2026 15:46:00 GMT

TL;DR

Playwright v1.60 makes failure evidence easier to capture during the run.

The main change is scoped HAR recording.

HAR means network request file.

It shows what the browser sent and received.

The release also adds file drops, ARIA boxes, and hard test aborts.

ARIA means accessibility map.

Together, these changes help CI failures explain themselves.

CI means automated build server.

The Release

Playwright v1.60 shipped on May 11, 2026.

The headline change is context.tracing.startHar().

Tracing means run evidence capture.

Before v1.60, HAR capture lived outside that tracing flow.

Now HAR recording sits next to other test proof.

The API also returns a Disposable.

A disposable is a cleanup handle.

You can use await using to close the HAR automatically.

That matters when many tests run at once.

Manual cleanup breaks easily.

One missed cleanup step can leave broken evidence behind.

Why This Matters For QA Teams

Most teams collect proof after a test fails.

That is too late.

The failing run is already gone.

The next run may pass.

Now the team has a guess, not proof.

Playwright v1.60 moves more proof into the first run.

That is the architectural part.

You are not asking a person to rerun the test.

You are designing the run to bring receipts.

That matters even more for AI testing.

AI means software that predicts.

Predictions need clear input.

A trace, a HAR file, and an ARIA snapshot give that input.

Without proof, AI just writes a confident guess.

How To Use Scoped HAR Recording

This example records network evidence for an upload test.

It also uses the new Drop API.

Drop API means file drop simulation.

import { test, expect } from '@playwright/test';

test('upload records network evidence', async ({ context }) => {
  await using har = await context.tracing.startHar('upload.har', {
    content: 'embed',
    mode: 'minimal',
    urlFilter: /\/api\/upload/,
  });

  const page = await context.newPage();
  await page.goto('/upload');

  await page.locator('#dropzone').drop({
    files: {
      name: 'note.txt',
      mimeType: 'text/plain',
      buffer: Buffer.from('hello'),
    },
  });

  await expect(page.getByText('Upload complete')).toBeVisible();
});

The HAR starts before the page opens.

The urlFilter keeps the capture focused.

The drop step sends an in-memory file.

When the test scope ends, Playwright finalizes the HAR.

No extra afterEach block is needed.

No custom try/finally block is needed.

That is the small win.

The bigger win is trust.

When upload fails, the HAR file shows the request.

Your team can inspect the failed run.

They do not need to recreate it from memory.

The Hidden Updates Worth Watching

The release is bigger than HAR.

Five smaller changes point in the same direction.

First, locator.drop() tests real upload zones better.

It accepts files, text, HTML, or URI data.

URI means web address.

Playwright sends dragenter, dragover, and drop events.

If the app rejects the drop, the method throws.

That is useful.

It tells you the app never accepted the file.

Second, ARIA snapshots can include boxes.

Boxes mean element positions.

The format is [box=x,y,width,height].

Coordinates are viewport-relative CSS pixels.

Viewport means visible browser area.

This gives AI tools a cleaner page map.

They can see what exists and where it sits.

Third, page-level ARIA assertions now exist.

You can assert the page body directly.

That removes one small wrapper step.

Small wrapper steps matter in generated tests.

Fourth, test.abort() stops bad runs early.

It works inside fixtures, hooks, and route handlers.

Fixtures mean shared test setup.

Use it when the test breaks a safety rule.

For example, a test might publish to a shared page.

That run should stop right away.

Fifth, BrowserContext now mirrors page events.

BrowserContext means a browser sandbox.

Framework owners can listen once per context.

They do not need listeners on every page.

That helps when one test opens many tabs.

The Gotcha Nobody Is Talking About

Only one HAR recording can run per BrowserContext.

That is not a problem.

But it is a design rule.

Do not start one HAR for login and another for upload.

Use separate contexts when you need separate captures.

Or keep one capture narrow with urlFilter.

The Drop API has another useful edge.

If dragover does not call preventDefault(), Playwright throws.

That sounds annoying.

It is usually the bug.

The app rejected the drop before the file arrived.

ARIA boxes have one more boundary.

They use viewport coordinates, not full-page coordinates.

If your test scrolls, account for that.

If your page uses frames, account for that too.

Migration Notes

Playwright v1.60 removes several old APIs.

Check these before upgrading a large suite.

Replace Locator.ariaRef() with locator.ariaSnapshot().
Remove the handle option from exposeBinding() calls.
Remove the logger option from browser connection calls.
Replace videosPath and videoSize with recordVideo.

There is also a CDP update.

CDP means browser control protocol.

connectOverCDP() now accepts noDefaults.

That matters when attaching to an existing Chromium browser.

Chromium means Chrome browser engine.

With noDefaults: true, Playwright leaves default context settings alone.

What This Changes In CI

This release does not make tests smarter by itself.

It makes test runs easier to review.

That is the part I care about.

Good QA architecture is not just more tests.

It is better evidence from each run.

For a small suite, v1.60 is a nice upgrade.

For a large CI suite, it is more important.

You can scope network captures.

You can test upload zones without custom browser hacks.

You can give AI tools page structure and element positions.

You can stop unsafe tests before they poison shared state.

That is not hype.

That is how test systems become reviewable.

Verdict

I would upgrade Playwright test projects to v1.60 this week.

Start with the evidence APIs.

Add startHar() where network failures waste review time.

Add locator.drop() where upload tests use custom events.

Add ARIA boxes where AI tools inspect pages.

Then review the breaking removals before merging.

The release is not about one huge feature.

It is about a better proof layer.

That is what AI QA architecture needs.

Related reads on anton.qa:

AI Test Automation Architecture: The 3-Layer System
Playwright release notes coverage from Anton's content plan

Official sources:

AI Test Automation Architecture: The 3-Layer System

Anton Gulin — Wed, 13 May 2026 23:32:00 GMT

AI test automation architecture is the system that tells AI what to test.

It also defines how to run tests and prove the result.

I split it into three layers: orchestration, execution, and evidence.

Without all three, AI testing becomes prompt output with no production gate.

Why tool lists fail

Most AI testing content starts with tools.

That is backwards.

AI means software that predicts.

Predictions can help QA teams move faster.

But predictions do not prove quality.

A tool can generate a test.

It cannot decide release risk alone.

It cannot prove the browser state was clean.

It cannot explain why a failure matters.

That work belongs to architecture.

The 3-layer model

I use three layers for AI test automation architecture.

Layer	Plain meaning	Main question
Orchestration	test control plan	What risk should this cover?
Execution	actual test run	Did it run in the real pipeline?
Evidence	proof from runs	Can a human review it?

If one layer is missing, the system gets weak.

If evidence is missing, the team gets false confidence.

Layer 1: Orchestration

Orchestration means test control plan.

This layer defines the work before AI writes anything.

It answers five questions:

What user flow matters?
What risk does this test cover?
What data must exist first?
What browser state is allowed?
What failure should block release?

AI can help draft the first version.

But a human still owns the risk call.

That is the difference between generation and architecture.

Layer 2: Execution

Execution means actual test run.

This layer proves the test can survive the real path.

That path is usually CI.

CI means automated build server.

A local demo is useful.

It is not enough.

Run the test where code ships.

Check browser state, cleanup, retries, test data, and worker isolation.

This is where Playwright and MCP matter.

Playwright is a browser test tool.

MCP means tool connection standard.

Together, they let AI agents use a live browser.

But the run still needs stable launch control.

That is why playwright-mcp v0.0.75 matters.

It serialized shared browser launch in isolated mode.

That means parallel runs get ordered startup.

Small release note.

Real architecture impact.

Layer 3: Evidence

Evidence means proof from runs.

This is the layer most teams skip.

Every AI-created test should leave receipts.

Useful receipts include:

trace
screenshot
log
video when timing matters
saved browser state when auth matters

The point is simple.

A reviewer should inspect the run without rerunning it.

If that is impossible, the test is not ready.

AI can write code quickly.

Review still needs proof.

A practical gate

Here is the gate I use before AI-generated tests ship.

Gate	Pass condition
Scope	The test maps to one named risk
Data	Test data setup is explicit
State	Browser state is controlled
Run	The test passes in CI
Evidence	Trace or equivalent proof exists
Review	A human can explain the failure mode

This is not heavy process.

It is a small guardrail.

It stops weak tests from becoming permanent debt.

What this changes for QA teams

The goal is not to slow AI down.

The goal is to make AI work reviewable.

When the architecture is clear, AI becomes useful in three places:

It drafts coverage ideas.
It writes first-pass test code.
It explains failures from evidence.

But humans still own the system.

Humans define risk.

Humans review evidence.

Humans decide what blocks release.

The rule

Never ask AI to expand test coverage first.

Build the proof system before that.

Generation is cheap.

Evidence is the architecture.

That is the line between AI testing demos and production QA.

Drag-and-Drop Automation is Finally Native in Playwright MCP

Anton Gulin — Wed, 06 May 2026 20:25:00 GMT

Drag-and-Drop Automation is Finally Native in Playwright MCP

I hate testing drag-and-drop.

Most QA teams do. We either skip it or we write hacks.

I have seen it all. Scripts that try to "flick" the mouse. Scripts that inject JavaScript.

They all break. They are hard to maintain.

This month, Playwright MCP v0.0.71 shipped a fix.

It adds a tool called browser_drop.

Why This Changes Things

Before, your AI agent had to guess how to move an item.

Now, it uses the native Playwright API: Locator.drop.

It works for:

Moving cards in a grid.

Dropping files into an upload zone.

Reordering lists.

It works because it handles the mouse events for you.

No more "faking" the move.

How to Use It

Here is one line of code to use in your MCP tool:

That is it. One command.

The AI agent finds the file. It moves it to the zone. It drops it.

The "Below the Fold" Gotcha

There is one small catch.

Playwright needs the items to be on your screen to drop them.

If your list is long and the target is at the bottom, the drop will fail.

The Fix: Scroll the target into view first.

This simple step stops your tests from failing in CI (where screens are smaller).

What Else Is New?

The v0.0.71 release added two more features:

Network Bodies: browser_network_requests now captures the full response from your server. You can check what the API said, not just the status code.

Plain Code: browser_evaluate now takes plain JavaScript. You do not need a function wrapper anymore.

Verdict

If you use AI agents for testing, upgrade today.

browser_drop removes the last big "hack" in our pipeline.

I architect the systems AI agents run on. Native tools like this make those systems stable.

How to Test MCP Servers Before They Break Your CI

Anton Gulin — Wed, 06 May 2026 10:00:00 GMT

How to Test MCP Servers Before They Break Your CI

Most teams install an MCP server and hope it works.

That is how you get 3 AM pages.

An MCP server is a bridge between AI agents and your tools. It can crash, leak data, or silently return garbage. If your AI agent relies on it, your whole pipeline breaks.

I have seen it happen. A team wired an untested file-system MCP server into a customer support agent. The agent deleted a config folder because the server had no path sandboxing. The fix was simple: test the server's permission model before wiring it in.

This post is that test. It is the checklist I run on every MCP server before it touches production.

What an MCP server is (in plain words)

MCP stands for Model Context Protocol. It is a standard way for AI agents to talk to external tools like browsers, databases, or file systems.

Think of it like a USB port for AI. The agent plugs into the server, and the server gives it access to a tool.

The danger is that the agent does not know what the tool can do. If the server has no guardrails, the agent can ask it to do anything: read files, delete data, call external APIs.

That is why we test the server, not just the agent.

The three-layer test stack

I test MCP servers in three layers. Each layer catches different failure modes.

Layer	What it catches	Tool
Discovery	Missing tools, broken metadata, wrong version	MCP Inspector
Behavior	Silent failures, wrong output, edge-case crashes	pytest smoke tests
Security	Over-permissions, data leaks, injection risks	Permission audit + static scan

If a server fails any layer, it does not ship.

Layer 1: Discovery with MCP Inspector

MCP Inspector is the official debugging tool for MCP servers. It is free and runs in your browser.

Start it with:

npx @anthropic-ai/mcp-inspector node dist/server.js

Then check these three things:

1. Does the server start without errors?

If the Inspector shows a red error on launch, the server has a dependency or initialization bug. Fix that first.

2. Does it list the tools it promises?

Open the "Tools" tab. Count them. Compare to the README. If the README promises 5 tools and the Inspector shows 3, the server is incomplete.

3. Does a sample request return the right shape?

Pick the simplest tool. Fire a request. Check that the response is JSON, has the right fields, and makes sense.

If the response is a plain string instead of structured JSON, your agent will fail to parse it.

Layer 2: Behavior with pytest smoke tests

MCP Inspector is great for manual checks. But manual checks do not scale. You need automated tests in CI.

Here is a minimal pytest test that initializes a server and verifies it responds:

import subprocess
import json
import pytest

@pytest.fixture
def mcp_server():
    proc = subprocess.Popen(
        ["npx", "-y", "@anthropic-ai/mcp-server-filesystem", "/tmp/mcp-test"],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        text=True
    )
    yield proc
    proc.terminate()
    proc.wait(timeout=5)

def send_message(proc, method, params=None):
    msg = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": method,
        "params": params or {}
    }
    proc.stdin.write(json.dumps(msg) + "\n")
    proc.stdin.flush()
    return json.loads(proc.stdout.readline())

def test_server_initializes(mcp_server):
    response = send_message(mcp_server, "initialize", {
        "protocolVersion": "2024-11-05",
        "capabilities": {},
        "clientInfo": {"name": "test", "version": "1.0"}
    })
    assert response["id"] == 1
    assert "result" in response

def test_tool_list_not_empty(mcp_server):
    send_message(mcp_server, "initialize", {
        "protocolVersion": "2024-11-05",
        "capabilities": {},
        "clientInfo": {"name": "test", "version": "1.0"}
    })
    response = send_message(mcp_server, "tools/list")
    assert "result" in response
    assert len(response["result"]["tools"]) > 0

Run this in CI. If the server fails to initialize, the build stops.

For Python-based servers, use FastMCP's in-memory testing. It runs without a subprocess:

from fastmcp import FastMCP
import pytest

mcp = FastMCP("test")

@mcp.tool()
def add(a: int, b: int) -> int:
    return a + b

def test_add_tool():
    result = mcp.call_tool("add", {"a": 2, "b": 3})
    assert result == 5

Layer 3: Security with a permission audit

Every MCP server requests permissions. Most teams click "allow all" and move on.

I check three things:

1. Does it need file system access? If yes, which paths?

A file server that requests / can read your entire disk. A good server requests a single folder, like /tmp/mcp-workspace.

2. Does it make network calls? To which hosts?

A server that calls api.github.com is fine. A server that calls any host is a data exfiltration risk.

3. Does it run shell commands? Under which user?

Shell access is the highest risk. If the server runs as root, any injected command can destroy the system.

I block these combinations:

File access	Network access	Shell access	Verdict
Single folder	None	None	OK
Single folder	Whitelisted host	None	OK
Any folder	Any host	None	Review required
Any folder	Any host	Yes	Block

The CI gate

After the three-layer check, I add one more step to CI: a gate script that runs all checks and outputs a report.

#!/bin/bash
# mcp-gate.sh — run before any MCP server ships

SERVER=$1
EXIT_CODE=0

echo "=== MCP Server Gate: $SERVER ==="

# Layer 1: Discovery
npx @anthropic-ai/mcp-inspector $SERVER --headless || EXIT_CODE=1

# Layer 2: Behavior
pytest tests/mcp-smoke/ --server=$SERVER || EXIT_CODE=1

# Layer 3: Security
node scripts/audit-mcp-permissions.js $SERVER || EXIT_CODE=1

if [ $EXIT_CODE -eq 0 ]; then
    echo "✅ Gate passed. Server can ship."
else
    echo "❌ Gate failed. Fix the issues above."
fi

exit $EXIT_CODE

Run this in CI. If it fails, the deployment stops.

Where to find servers worth testing

The MCP ecosystem moves fast. Here are the three places I look:

Official MCP Registry — https://registry.modelcontextprotocol.io
Microsoft now publishes Playwright MCP here. Any server in this registry has at least passed a basic review.
GitHub — Search modelcontextprotocol topics. Check last commit date and test coverage before installing.
npm / pip — Search @anthropic-ai/mcp-server-* or mcp-server-*. Read the README. Check the download count.

Red flags:

No commits in 6+ months
No tests in the repo
No README explaining what it does
Permission requests that are too broad

Verdict

Testing MCP servers is not optional. An untested server is a bug waiting to become an incident.

The three-layer stack — Discovery, Behavior, Security — catches the failure modes I have seen in production. MCP Inspector for manual checks. pytest for CI gates. A permission audit for the last line of defense.

Playwright MCP ships often and is published in the official MCP Registry. If you are building AI test infrastructure, start there. It has the audit trail and the active maintenance that production work requires.

Start with one server this week. Run it through the checklist. That is how you build AI infrastructure that does not wake you up at 3 AM.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, now Lead Software Engineer in Test. Find him at anton.qa or on LinkedIn.

Playwright MCP v0.0.73: How to Configure Browser Paths via Environment Variables

Anton Gulin — Sat, 02 May 2026 21:19:00 GMT

TL;DR

Playwright MCP v0.0.73 fixes a critical gap where extension channels and executable paths couldn't be resolved from CI/CD environment variables. This release enables true containerized test pipelines—configure browser installations once in your Dockerfile, override per job via environment variables, no hardcoded paths required.

The Release

Playwright MCP v0.0.73 shipped two interconnected bug fixes that resolve a pain point I have seen break containerized test pipelines at scale.

The first fix resolves extension channel and executablePath from CLI flags and environment variables (#40572). The second propagates --browser channel flags on the --extension path (#40567). Combined, these changes mean your Playwright MCP setup can now be fully environment-driven—critical for organizations running tests in Docker, Kubernetes, or ephemeral CI workers.

Additionally, the package is now published to the official MCP Registry, which simplifies enterprise procurement and governance for teams evaluating AI-assisted testing infrastructure.

Why This Matters for Engineers and QA

If you are running Playwright in containers, you have probably hit the hardcoded path problem. You install Chromium in your Dockerfile at /usr/bin/chromium-browser, but your CI system runs on macOS runners where the path is /Applications/Chromium.app, and your staging environment uses a custom Brave installation somewhere in /opt/brave. Without environment variable support, you end up maintaining per-environment Dockerfiles, CI configs with conditional logic, or wrapper scripts that fragment your test infrastructure.

This is not a theoretical problem. I have seen teams abandon containerized Playwright testing entirely because the path resolution complexity outweighed the benefits of ephemeral environments. The fix in v0.0.73 closes that gap.

How to Use It

With this release, you can configure browser channels and executable paths through environment variables, making your Playwright MCP configuration portable across environments. Here is the working pattern:

Or in your Node.js test runner:

For CI/CD pipelines, the pattern becomes environment-agnostic:

The --browser channel now propagates correctly when --extension paths are specified, meaning parallel test workers can target different browsers without configuration conflicts.

The Gotcha Nobody Is Talking About

Here is the honest gotcha: this fix only works if your environment variables are being read by the Playwright process, not the MCP server subprocess. If you are running Playwright MCP as a standalone server (common in AI-assisted testing setups), environment variables set in your shell may not propagate to the MCP process spawned by your AI tool.

Test this before assuming it works:

If your AI coding assistant spawns the MCP server, you may need to pass environment variables through the MCP configuration file or your AI tool's environment settings, not just the shell.

What This Changes in Your CI Pipeline

With v0.0.73, you can standardize your Playwright MCP configuration across all environments. The path resolution hierarchy is now:

CLI flags (highest priority)

Environment variables

Config file defaults

Built-in channel defaults

This means your Docker base image can ship with a sensible default browser path, your CI jobs override it via environment variables, and your local development uses the config file. No more per-environment Dockerfile maintenance.

For teams targeting faster deployments, this also removes a class of environment-specific debugging that typically consumes 15-30 minutes per incident. The 50% faster deployment metric I have seen at CooperVision includes eliminating these friction points from CI pipelines.

Verdict

Playwright MCP v0.0.73 is a quiet but significant release for teams running containerized test infrastructure. The MCP Registry publication is nice for enterprise discoverability, but the bug fixes around environment-driven browser resolution are the substantive change. If you are managing Playwright across multiple environments, update now and audit your environment variable configuration to take full advantage of the new resolution order.

The gotcha about subprocess environment propagation is real—test your specific setup before deploying to production. Environment variables that work in your shell may not propagate correctly through MCP server spawns in AI tooling contexts.

https://x.com/aiwithanton/status/2051411902216163582?s=20

For full release details, see the official release notes.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test at CooperVision. Find him at anton.qa or on LinkedIn.

browser_run_code_unsafe: The Naming Change That Forces Honest Architecture

Anton Gulin — Thu, 30 Apr 2026 23:46:00 GMT

TL;DR

Playwright MCP v0.0.72 renamed browser_run_code to browser_run_code_unsafe, making sandbox escape risk explicit. The new browser_network_request tool enables indexed request inspection. Teams using this MCP server in CI pipelines need to update tool references and security documentation now.

The Release

Microsoft shipped playwright-mcp v0.0.72 on April 30, 2026 with two headline changes: a renamed tool that forces explicit acknowledgment of security boundaries, and a new indexed network request API.

The tool formerly known as browser_run_code is now browser_run_code_unsafe. The rename surfaces a capability that was always there—arbitrary code execution within the Playwright browser context—and makes the sandbox implications impossible to miss in code reviews, CI configs, and architecture docs.

Also new: browser_network_requests now returns a numbered list, and browser_network_request lets you fetch any single request by index, including headers and body. Both tools accept a filename option for direct file output.

Why This Matters for Engineers, QA, and CI

I've seen what happens when security assumptions live in the gap between "works in dev" and "approved for prod." At CooperVision, where I've led test infrastructure through a 40% faster test execution migration from Selenium to Playwright, the gap between a tool's name and its actual risk profile costs real time.

When a tool is named browser_run_code, it's easy to approve it in a security review without asking what code actually runs. When it's named browser_run_code_unsafe, you can't pretend the sandbox boundary is theoretical. This rename is documentation embedded in the API contract.

For QA architects integrating MCP servers into CI pipelines, this matters in three places: security review templates, tool permission configs in your MCP client, and any internal docs that reference the old name.

How to Use It

The new indexed request API is straightforward. Here's a TypeScript example using the new browser_network_request with an index:

For the renamed tool, update your tool references:

The functionality is identical. The name changed.

The Gotcha Nobody Is Talking About

The rename doesn't change behavior—but it will break your existing integrations silently if you're using tool name matching in your MCP client config. If you hardcoded browser_run_code in a tool registry or a Claude Desktop config file, you'll get a "tool not found" error with no explanation that a rename is the cause.

The unhandled rejection fix (previously crashing the server on unhandled errors in browser_run_code_unsafe) means old error patterns in your logs may disappear. That's good, but if you had monitoring that expected crashes on certain error types, you'll need to update alert thresholds.

What This Changes in Your CI Pipeline

If you're running Playwright MCP in CI—using it to drive browser automation for end-to-end tests or visual regression checks—the rename forces a config audit. In practice, this is a one-line change in your tool mapping, but it should trigger a broader review of what other tools you're calling and whether their names still match upstream.

The new filename option for network tools solves a real CI artifact problem: network logs that used to need in-memory buffering and post-processing can now write directly to the filesystem, making them available for artifact upload without extra code.

At CooperVision, I've shipped changes that cut deployment times in half partly because artifact handling got smarter. Direct file output from MCP tools is that kind of small win that compounds across hundreds of test runs.

Migration Notes

Update all tool name references from browser_run_code to browser_run_code_unsafe

Review security documentation to reflect the explicit "unsafe" designation

Check MCP client configs for hardcoded tool names

The filename parameter is backward-compatible (optional, existing behavior unchanged)

The indexed request API is additive; existing browser_network_requests behavior works as before

Verdict

The browser_run_code_unsafe rename is the right call. It's an API contract change that forces honest conversations about what your automation tools can actually do. In enterprise environments where QA architects own security review sign-off, explicit naming prevents the kind of assumption drift that creates vulnerabilities.

The new indexed network request API is a solid ergonomic improvement—pinpoint access to individual requests without parsing full lists. Combined with the filename output option, it makes Playwright MCP a more capable tool for CI pipelines that need artifact generation.

This is a non-breaking change for functionality, but a breaking change for tool name references. Audit your configs, update your docs, and move on. The 40% faster test execution gains from the Selenium→Playwright migration taught me that staying current on toolchain changes pays off—it's rarely as painful as it feels in the moment.

Upgrade and update your tool names. Then look at what you can do with indexed request inspection that you couldn't before.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test at CooperVision. Find him at anton.qa or on LinkedIn.

Playwright CLI v0.1.10 Brings Spec-Driven Testing Skills for AI Agents

Anton Gulin — Thu, 30 Apr 2026 23:39:00 GMT

TL;DR

Playwright CLI v0.1.10 introduces a spec-driven testing skill that guides AI agents through plan/generate/heal workflows for maintaining test suites from written specifications. Network inspection now uses stable request indexing, and raw output is default for all data-fetching commands—eliminating preprocessing steps in CI pipelines.

The release

Playwright CLI v0.1.10 shipped on April 30th with two headline features that reshape how AI agents interact with browser automation. The network inspection subsystem got a complete overhaul—network is gone, replaced by requests plus granular subcommands that output indexed, pipe-friendly data. But the feature I'm actually excited about is the spec-driven testing skill: a references/spec-driven-testing.md reference that teaches AI agents to drive Playwright tests from written specifications.

This matters because I spend my days building test infrastructure that scales. At CooperVision, I oversaw a 300% increase in test count while hitting 50% faster deployments. That didn't happen by writing more tests manually—it happened by building workflows that let the system do the repetitive work. The spec-driven testing skill is exactly that kind of workflow accelerator.

Why this matters for engineers and QA architects

If you're running AI-augmented QA workflows, you know the problem: agents can generate test code, but without a structured pattern for keeping that code alive, regressions pile up and the suite rots. The spec-driven testing skill provides that structure.

The references/spec-driven-testing.md file outlines a plan/generate/heal cycle. An agent reads a written spec, plans which Playwright assertions map to the spec's behavior, generates the corresponding test code, then heals regressions when the spec changes or the app drifts. This isn't hypothetical—I've seen this pattern work at scale when migrating from Selenium to Playwright, where we achieved 40% faster test execution.

For CI engineers, the network inspection overhaul matters more immediately. The old network command inlined bodies and required brittle string parsing. The new numbered commands (requests, request , request-headers , request-body ) output stable indexes and pipe-friendly data. You can pipe directly to jq without stripping wrapper text.

How to use it

Here's the spec-driven testing workflow in practice. The skill lives at references/spec-driven-testing.md and gets loaded automatically when you're running Playwright CLI with agent-mode enabled:

Start Playwright CLI with agent mode

npx playwright-cli --agent

For the network inspection overhaul, here's a working example:

The raw output change is significant: data-fetching commands like cookie-list, localstorage-list, and route-list now emit unwrapped output by default. Your CI scripts drop 1-2 preprocessing steps per invocation.

The gotcha nobody is talking about

The spec-driven testing skill is read-only in this release. You get the reference file and the pattern, but there's no built-in mechanism for the agent to automatically detect spec changes and trigger heals. You have to wire that yourself. In my experience, the first implementation is always manual—you define the trigger conditions, the diff logic, the heal policy. The skill gives you the pattern; you build the automation.

This is honest: don't expect autonomous test maintenance out of the box. Plan for 2-4 weeks of integration work to wire the heal step into your CI pipeline, depending on your test suite size and the stability of your application contract.

What this changes in your CI pipeline

Three concrete changes:

Network data is now scriptable. You can pull request headers, bodies, and responses directly into CI steps without parsing wrapper text. I estimate this saves 15-30 minutes of scripting work per team per quarter.

Spec-driven test maintenance becomes possible. If you're running AI agents for QA, the pattern exists now. You still need to implement the trigger/heal logic, but the framework is there.

MCP server stability improves. Unhandled promise rejections from user callbacks no longer crash the transport. If you're running Playwright CLI as a long-running MCP server (common in AI-augmented workflows), this is meaningful reliability.

Migration notes

The network command is replaced. If you have scripts that parse network output, they need updating:

Data-fetching commands (cookie-list, route-list, etc.) now emit raw output. If you were stripping the ### Result wrapper, remove that logic—it's gone.

Config-relative path resolution for initPage and initScript now resolves against the config directory (matching Vite/Vitest/ESLint behavior). If you were working around silent load failures, those workarounds are no longer needed.

Verdict

Playwright CLI v0.1.10 is a meaningful release for AI QA architects and CI engineers alike. The spec-driven testing skill is the headline for teams adopting AI-augmented workflows—it provides the pattern, even if the automation is still DIY. The network inspection overhaul is the practical win for everyone else: stable indexing, pipe-friendly output, raw by default.

My recommendation: upgrade, wire the MCP server stability fixes first (low risk, immediate benefit), then evaluate the network inspection changes for your CI scripts. The spec-driven skill is a longer-term investment—budget the integration time honestly, and you'll get the payoff.

If you're already running Playwright in your CI pipeline, this release makes it easier to script and maintain. If you're building AI-augmented QA workflows, the skill gives you a structure to teach your agents. Both audiences win.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test at CooperVision. Find him at anton.qa or on LinkedIn.

The MCP Ecosystem Just Collapsed Into Playwright

Anton Gulin — Thu, 30 Apr 2026 00:32:00 GMT

I spent years fighting login scripts for tests.

AI agents made it worse. They still get stuck at login screens. This month, that changed.

MCP (how AI talks to tools) is changing how we test. Two teams at Microsoft shipped a fix. They released two ways to share your browser with AI.

Playwright core added browser.bind().

The CLI team added an MCP Bridge.

Both land in the same month. This is not a test. It is the new way.

The Drama: What This Replaces

Browser tests have a "login gap." Headless browsers (browsers without a window) start empty. They have no cookies, sessions, or secure login codes.

When your app needs a company login, a headless browser fails. Teams solve it with hacks. We export cookies, make "special" accounts, or snapshot Docker images.

I spent 20% of my time just keeping these hacks alive. These new tools don't solve login. They skip it. The human logs in once. The agent uses the same browser.

What browser.bind() Does

browser.bind() shares the browser you are already using. It shares your cookies and your tabs. When you call it, Playwright creates a local data connection.

Any AI agent connects to that pipe. It sees the exact same browser you see.

This is for developers. One line of code. No more re-logging in. No more cookie scripts.

What the MCP Bridge Extension Does

The extension is the "no-code" path. Install it in Chrome and click one button. Your open tab now talks to your AI.

Claude (the AI) sees the page. It can click and type. No terminal needed. Just one click.

Before: Login -> export cookies -> start new browser.
After: Click the extension. Done.

Both tools shipped in one month. They solve the same pain.

The Architect's Read: A New Standard

This is a standard being born. Before, every team built their own bridge to the browser. Now, it is a standard feature.

Two teams shipped one pattern in 30 days. The signal is clear. The browser is now a shared space. Humans and AI agents work on the same surface.

What does this mean for QA? It means you stop babysitting login scripts. Your AI agent tests run where your humans test. The wall between "human" and "agent" testing is gone.

What Most Teams Miss

The win is not just a faster script. The win is "Shared State." Before, we had parallel worlds. One human click and one agent click meant two browsers and two truths.

Now they are one. When an AI agent finds a bug, you open the same browser. You see the same state and same session.

No more "it works on my machine." Same browser. Same bug.

How to Try It Today

If you use Playwright, browser.bind() is in v1.59.0. If you want the no-code path, get the extension. Then run playwright-cli attach --extension.

Both connect your agent to a real, logged-in browser. Not a toy. Use the browser your team uses.

Native Drag-and-Drop Automation Arrives in Playwright MCP: What v0.0.71 Changes

Anton Gulin — Tue, 28 Apr 2026 18:25:00 GMT

TL;DR

Playwright MCP v0.0.71 ships browser_drop. It gives you native drag-and-drop from any MCP client. No more evaluate scripts. No more mouse.move chains. Grid reordering, file drop zones, text editor drags — all work the same way a real user does.

Why This Release Matters

QA teams either abandon drag-and-drop testing or hack around it. But sortable grids, file uploads, and rich text editors are everywhere. And they have been painful to test forever.

I ran into this firsthand on one project. Solid Playwright coverage for clicks, typing, and navigation. But drag-and-drop? We used evaluate scripts. Or we tested it by hand. Both paths broke across browsers. Both were impossible to keep working.

Playwright MCP v0.0.71 fixes this with browser_drop. It uses Playwright's own Locator.drop — the same API your tests already use. Now any MCP client can call it.

How to Use browser_drop

Here's a complete example combining browser_drop with the new response body capture from browser_network_requests and the simplified expression support in browser_evaluate. This pipeline automates a file upload scenario, validates the server response, and confirms the UI state update:

Three new tools working together: browser_drop handles the drag. browser_network_requests captures the server response (full body, not just status codes). browser_evaluate runs plain JavaScript — no function wrapper needed.

The Gotcha Nobody Is Talking About

browser_drop needs both elements to be on screen. That's correct Playwright behavior. But here's the catch: if you navigate to a page and the drag target sits below the fold, the drop fails.

The fix: Call browser_evaluate to scroll the target into view before calling browser_drop, or use the scroll option if your Playwright version supports it. This catches teams off guard in CI where viewport sizes are smaller than local development.

This is not a bug. It's how Playwright works. But it catches teams when they test on a big screen and deploy to CI. CI viewports are smaller. The element you tested locally is off screen in the pipeline.

What This Changes in Your CI Pipeline

With browser_drop, you can test drag-and-drop flows through MCP. Not by hand. Not with broken scripts.

On one project, Selenium to Playwright gave us 40% faster tests. But drag-and-drop still broke in headless mode. We wrote evaluate scripts. They stopped working every sprint. browser_drop puts native drag-and-drop into MCP. No scripts. No workarounds.

What this actually means:

Fewer flaky tests. Native drag-and-drop is tested across browsers. evaluate + mouse.move sequences are not.

Simpler AI test generation. AI tools call browser_drop directly. No fragile mouse chains.

Faster CI. Native operations run faster than JavaScript-injected drag scripts.

Verdict

Playwright MCP v0.0.71 is worth upgrading for browser_drop alone. The response body capture and plain expression support make it better. But drag-and-drop was the missing piece. Now it's there.

The catch is real but small. Scroll your target into view before you drop. One line. Add it to your tool definitions and move on.

If you run MCP-based test infrastructure, this kills the last reason to fall back to evaluate for drag-and-drop. Upgrade. Add the scroll guard. Ship.

Reference: Playwright Locator.drop API documentation

Playwright Just Shipped the Fix For Flaky Tests I Built 3 Years Ago

Anton Gulin — Thu, 23 Apr 2026 17:12:00 GMT

Three years ago, I built a test framework that fixed itself. Nobody called it AI. "Agent" was still the thing antivirus ran on your laptop.

I split the framework into three parts. I called them Planner, Generator, and Healer. Not because I read a paper. Those were the three jobs I needed. I was out of good names.

Last October, Playwright shipped three Test Agents in version 1.56. Three of them.

They call them Planner, Generator, and Healer.

This month, version 1.59 shipped the rest of the plumbing. It added video recording inside tests (page.screencast). It added browser.bind(), so Claude or Cursor can connect to a running browser. It added async disposables (auto-cleanup for test resources). The agents shipped in October. Their plumbing shipped last week.

So this post is about one thing. The same three-part system that saved my career just shipped as a feature in the tool everyone uses.

Here is what Playwright got right. Here is what is still missing. And here is how to start using it today. Even if you stay on your own framework.

If your tests fail at random, and someone keeps asking you to "just make the flaky tests pass" — read this.

The Problem: the flaky-test cost nobody budgets for

Here is a cost every engineering manager forgets: the flaky-test cost.

One team I worked with had 1,200 end-to-end tests. About 4% failed at random on each run. Sounds small. It was not.

4% random failures × 20 PR runs a day = ~1,000 fake failures a week

Every fake failure starts a re-run, a check, a Slack thread

On a good week, 3 engineers each lost a day to false failures

On a bad week, it took the whole team for two sprints

That is the flaky-test cost. It costs you people, not money. That is why budgets miss it. It shows up as missed deadlines, canceled demos, and tired engineers.

The normal fix is "try harder."

Better locators (how tests find buttons on the page)

Wait on the right event

Don't trust the backend

Park the bad tests in quarantine

Review the quarantine every week

All true. None is enough. You can try harder. Flaky tests keep growing.

So I stopped fixing each test. I started fixing how all tests work together.

The Drama: two weeks that broke me

I won't name the company. I will say this. My tests passed on my laptop. They failed only on clean CI builds. They failed when they ran beside another team's tests.

Sometimes they failed. Not every time. Always on Tuesday, between 10:14 AM and 10:22 AM.

We lost two weeks. I tried everything. I tried everything again. I tried everything in a new order.

On day 11, I stood at a whiteboard at 9 PM. The board was full of arrows. I finally saw the truth.

The tests were fine. The framework was the problem.

My framework thought the app was the only thing under test. It was not. The CI server was under test too. So was the database snapshot job. So was the deploy timing on the staging server.

We fixed that one bug. But the two weeks taught me the big lesson:

Fixing flaky tests is not a writing problem. It is a design problem.

The tests don't need more rules. The framework around them needs to be smarter.

That is where the three-part system was born.

The Solution: Planner, Generator, Healer

Here is the whole system in short. The names are mine. The ideas are obvious once you stop pretending they are one job.

Planner

Job: read a feature, a user story, or a bug. Write a test plan.

Not code. A plan. A list of flows, edge cases, set-up, clean-up. In plain Markdown.

Why it is its own job: planning and writing are not the same skill. If one thing does both, tests drift from the plan. You get tests the agent can't explain. And gaps where it had no example to copy.

Plan first. Write later.

What I built three years ago: a plan generator that read from PR descriptions, Jira tickets, and production alerts. It produced a Markdown plan. Engineers reviewed it before any code was written. About 85% of plans were approved as-is. The 15% that were rejected were caught in minutes. Not days of debugging.

Generator

Job: take an approved plan. Write the test code. Pick the button names. Write the checks. Set up the test.

Why it is its own job: code writing works best with a narrow goal (one plan). Not a wide goal (the whole codebase). A focused generator with one plan beats a smart generator with the whole repo.

What I built: a generator that turned plan Markdown into Playwright tests in TypeScript. It picked button names in a fixed order (data-testid first, then role, then text as last resort). It set up fixtures. It used soft checks by default. No creativity. One plan in, one test file out.

Healer

Job: a test fails. Check why.

Is it a real bug? A button name that moved? Or a slow server that day?

Fix the things you can. Flag the real bugs. Park the rest with notes.

Why it is its own job: and this is the part no one wanted to hear. Healing is not "run it again until it passes." That is hiding. Healing is three steps: check, propose a fix, get it reviewed.

What I built: a Healer that compared the current page to the last green run. If the button name was stale, it proposed three new candidates. It scored each one. It opened a pull request with the best one-line change. A human reviewed it.

Humans merged about 80% of those fixes. The other 20% were caught in review. That is exactly what a good Healer looks like.

100% merged means humans aren't reading

20% merged means the Healer is broken

80% merged means both sides are working

The Numbers

I don't love numbers without a shop name. My rules don't let me name the shop. So here is what I can tell you plainly:

On one project, the three-part system let the test suite grow 3× in 18 months. The flaky-test rate stayed flat.

On another, each engineer spent a third less time on broken tests in the first quarter.

On a third, one CSS rename broke 100+ tests overnight. The Healer fixed it in one pull request by morning. The old way was a 3-week cleanup.

These numbers are not magic. They come from splitting the work into three small jobs. And from watching the handoff between each job. If you already do this with your services, you already know why it works.

Now Playwright Ships This As A Feature

Playwright versions 1.58 and 1.59 shipped a set of Test Agents in VS Code and the command line:

Planner agent — explores the app, writes test plans

Generator agent — turns plans into test code

Healer agent — fixes failing tests with AI help

The release notes: v1.58 and v1.59. The agent APIs are browser.bind() and page.screencast.

Same three jobs. Same split. Microsoft built what I built. They built it better in some ways. They missed one big thing.

What Microsoft got right

Each agent works alone. You can run Planner by itself. Pass its output to Generator. Never touch Healer. That split is the whole point. An agent system where everything is tangled is just one big prompt.

The agents are optional. You don't have to buy in all at once. Drop the Healer into your old tests. Leave Planner and Generator for later. That is how real teams adopt new tools.

They shipped the plumbing, not just the agents. Two pieces matter:

browser.bind() — added in v1.59. It lets any AI tool like Claude or Cursor connect to a running browser. No fresh browser. No lost cookies. No mocking your login.

Playwright MCP Bridge — a free Chrome extension. It connects your open tabs to a local Playwright server. Your real cookies. Your real profile. Your real logged-in session.

Together, those two things solve a problem QA teams have been hacking around for years. Let an AI agent work on your real browser. Not a fresh empty one. Microsoft built the plumbing. You don't have to.

What Microsoft missed

The review loop.

Self-healing is not a feature. It is a deal between the test, the app, and the team.

The Healer will happily propose fixes. But who reviews them? Who sets the merge rules? Who steps in when the Healer's fix rate drops? Playwright ships the agent. It does not ship the rules around the agent.

Those rules are the hard part. And you have to build them. Whether you use Microsoft's agents or your own.

A Healer with no review loop is just a bug generator with a nice screen.

If you're on Selenium, Cypress, or something older, the migration math got better with v1.59 this month — but the pattern is portable. You don't need Microsoft's implementation to build this. You need:

Plans as artifacts. Markdown. Version-controlled. Reviewable.

Generators with narrow context. One plan in. One test file out. No repo-wide reasoning.

A healer with a review loop. It proposes, a human approves, CI enforces. If the human always approves, your healer is working. If the human always rejects, your healer is broken. If it's 80/20, it's doing its job.

What to Do Next (Even if You Don't Migrate)

If you already use Playwright, the path is simple. Try the Planner agent in VS Code next sprint. Feed it one real user story. Compare its plan to your plan. Do that 10 times. If you would hand its plans to a junior engineer, it works. That means you found a 2–3× speed boost.

If you use Selenium, Cypress, or something older, migration got easier this month. But the system is portable. You don't need Microsoft's tools to build it. You need three things:

Plans as files. Markdown. In git. Reviewable.

A generator with a narrow goal. One plan in. One file out. No repo-wide thinking.

A Healer with a review loop. It proposes. A human approves. CI enforces. 80% merge rate means it works.

Start with the Healer if flaky tests block releases. Start with the Planner if you are short-staffed. Start with the Generator last. It is the flashy one. But it is the least useful without the other two.

If your team doesn't have this yet, print this post. Paste it in your design doc. Replace "I built" with "we can build." Take it to your next architecture review.

The Takeaway

Three years ago, this system was a weird thing a weird architect built. Nothing off the shelf solved the problem.

This month, it ships as a native feature in the tool serious web teams use. Last October, the agents shipped inside Playwright. This week's v1.59 release added the production parts: video receipts, MCP interop (AI tool bridge), and async disposables.

If you are still treating flaky tests as a writing problem, you are three years behind.

If you treat them as a design problem, you are on time. The pattern worked then. It ships natively now — agents in v1.56, infrastructure in v1.59. The contract around it is still yours to build.

If you have been treating them as a design problem for years, you are ahead of the team that ships the framework.

That is a fine place to be.

The system worked then. It ships in v1.59 now. The rules around it are still yours to build.

That is the job.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test systems where AI agents and human engineers work together on quality. Former Apple SDET (Apple.com and Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

Eval-Driven Development for AI Agent Skills

Anton Gulin — Sun, 19 Apr 2026 20:27:00 GMT

The Problem with Writing Skills by Hand

You've written a skill for your AI coding agent. It's got clear instructions, proper formatting, a good description. You test it in a session — it works. Ship it, right?

Not so fast.

Skills trigger based on their description field — a 1-2 sentence summary in the SKILL.md frontmatter. And here's the thing: descriptions that seem crystal clear to humans often trigger wrong. Too specific, and the skill never activates when it should. Too broad, and it fires on unrelated prompts.

The result: skills that feel right in theory but fail unpredictably in practice. And there's no systematic way to measure whether a skill is getting better or worse across iterations.

This is the same problem software engineering solved decades ago with automated testing. Skills are software. They need testing too.

What Is Eval-Driven Development?

Eval-driven development is the practice of:

Writing test cases that define expected behavior

Running those tests automatically to measure actual vs. expected outcomes

Using the results to improve iteratively, with quantifiable evidence

For AI agent skills, this means:

Generating test prompts (should-trigger and should-not-trigger queries)

Running each prompt with and without the skill

Comparing outputs to see if the skill actually improves results

Optimizing the description so the skill triggers on the right prompts

The Skill Creation Lifecycle

opencode-skill-creator implements eval-driven development as a structured lifecycle:

Create → Evaluate → Optimize → Benchmark → Install
↑ |
└─── Iterate ──┘

Create

Start with an intake interview. The skill-creator asks 3-5 targeted questions:

What should this skill enable the agent to do?

When should it trigger?

What output format is expected?

What workflow steps must be preserved exactly?

This captures intent before writing any code.

Evaluate

Auto-generate eval test sets — realistic prompts categorized as should-trigger or should-not-trigger. Run each test case twice:

With skill: The agent has the skill loaded

Without skill: The agent runs without it (baseline)

This measures whether the skill actually improves the output for relevant prompts.

Optimize

The description optimization loop treats triggering accuracy as a search problem:

For each iteration (up to 5):

Evaluate current description on train set (60%)
Analyze failure patterns
LLM proposes improved description
Evaluate on both train AND test (40%) sets
Select best description by test score

The 60/40 train/test split prevents overfitting. An description that works perfectly on train queries but fails on held-out test queries is overfit — it's memorized specific prompts rather than learning the general pattern.

Benchmark

Run the full eval suite across multiple iterations with variance analysis. This answers:

Is the skill getting consistently better?

Are there eval cases where the skill never triggers correctly?

How much variance is there across runs?

The benchmark includes:

Pass rates (with-skill vs. baseline)

Timing data (tokens, duration)

Mean ± standard deviation for each metric

Install

Install the final validated skill to project-level (.opencode/skills/) or global (~/.config/opencode/skills/). Only the final version gets installed — eval artifacts stay in the staging directory.

Why This Works

Skills are software

They have inputs (prompts), outputs (agent behavior), and a triggering mechanism (the description). Just like any software, they need testing.

Manual testing doesn't scale

You can test a skill manually in a session, but that's one prompt, one run, no measurement. Eval-driven development gives you 20+ test cases, multiple runs per case, and quantitative metrics.

Description optimization is more impactful than skill content

The description field is the primary triggering mechanism. A perfectly-written skill with a poor description won't trigger. An average skill with an optimized description will trigger reliably. The optimization loop focuses effort where it matters most.

Train/test splits prevent overfitting

If you only test on the same queries you optimize for, descriptions become overfit — they work on those specific prompts but fail on real-world usage. The 60/40 split keeps you honest.

Human review catches what automation misses

The visual eval viewer puts outputs side by side so you can see with your own eyes whether the skill is producing good results. Quantitative metrics tell you if it's triggering correctly; human review tells you if the output is actually useful.

Getting Started

npx opencode-skill-creator install --global

Then ask OpenCode to create or improve a skill. The eval-driven workflow starts automatically.

Apache 2.0, free, open source. Works with any of OpenCode's supported models.

GitHub: https://github.com/antongulin/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global

Create Video Receipts for AI Agents with Playwright Screencast API

Anton Gulin — Fri, 17 Apr 2026 02:13:50 GMT

TL;DR

Playwright v1.59.0 ships the Screencast API, letting AI agents produce verifiable video evidence of their work. Engineers can replay agent actions with chapter markers and action annotations—no manual test replay required. Setup is three lines: start the screencast, run your agent logic, stop and save. This is the observability layer agentic workflows have been missing.

The Release

Playwright v1.59.0 dropped last week and the headline feature is the Screencast API. Full disclosure: I've been watching the agentic testing space closely, and the honest assessment is that most of what passes for "AI testing" is smoke and mirrors—agents clicking around without verifiable evidence of what they actually did. The Screencast API is different. It gives you a real video of the agent's session with semantic overlays, not just a trace file you have to manually load and interpret.

The API surface is straightforward: page.screencast.start() initiates recording and page.screencast.stop() finalizes it. Between those calls, Playwright captures JPEG frames in real-time and lets you annotate them with chapter titles and action labels. You get a video file you can attach to a ticket, drop in a Slack thread, or store as audit evidence.

This release also includes browser.bind() for MCP integration, a CLI debugger, and async disposables—but for this post, I'm focusing on the Screencast API because it's the feature that directly addresses the verification problem in agentic workflows.

Why This Matters for Engineers and QA

If you're building or evaluating AI coding agents that interact with browsers, you face a fundamental trust problem. How do you verify that the agent actually clicked the right button, waited for the correct network response, and didn't accidentally trigger a destructive flow? Logs help, but they're not persuasive in a code review. Screenshots help more, but they don't capture temporal sequences well.

Video receipts solve this. You get a playback of the full session with chapter markers at key decision points. Your PM can watch a 90-second clip instead of reading 200 lines of trace output. Your security team gets evidence they can archive. Your CI system gets an artifact to attach to the test report.

For QA teams specifically, this changes the audit story. When a flaky test gets investigated, you currently spend 20-30 minutes reproducing the environment, loading traces, and reconstructing what happened. With a screencast, you open a video. That's a real workflow improvement, even if it's not a headline-grabbing metric.

How to Use It

Here's the implementation. The API supports chapter titles, action annotations, and visual overlays. You can configure frame capture rate and output format.

The captureFrame() method is what makes this useful for AI vision workflows. You pass the JPEG buffer to your vision model for validation or further processing. The agent produces the evidence; you decide what to do with it.

The Gotcha Nobody Is Talking About

Here's what the release notes don't emphasize: screencast recording in headless mode is not pixel-perfect. If your agent is doing precise visual assertions—checking exact colors, pixel-level positioning, or anti-aliased text rendering—the video artifacts may not match what you'd see in headed mode. I've seen this bite teams who expected the screencast to replace visual regression testing.

The API works correctly and the implementation is solid, but it's recording a compressed video, not a pixel-accurate capture of the render pipeline. Use it for workflow verification, not for asserting that #FF5733 exactly matches your design token. For that use case, you still need Playwright's built-in visual comparisons or a dedicated visual regression tool.

Also worth noting: the output file can get large quickly. A 5-minute session at 15 fps with visual overlays will easily be 50-100MB. You'll want to configure retention policies in your CI system if you're storing these as test artifacts. Don't let this become your next storage incident.

What This Changes in Your CI Pipeline

The immediate impact is on how you handle failures from AI-driven test agents. Currently, when an agent-authored test fails, you have two options: trust the agent's explanation (risky) or manually reproduce the failure (slow). With screencasts, you get a third option: watch the video, verify the agent's logic, and make an informed decision in under 60 seconds.

In practice, this means fewer "cannot reproduce" situations in your backlog. The debugging loop tightens from hours to minutes. For teams running autonomous agents in CI—yes, that's a real thing—this is a meaningful improvement in the feedback cycle.

Storage considerations aside, the integration is straightforward. Add page.screencast.start() to your fixture setup, route failures to your screencast storage, and update your test reporters to embed video links. Your team will adapt faster than you expect.

Migration Notes

No migration required for existing tests. The Screencast API is additive—if you're not calling page.screencast.start(), your current suite is unaffected. The breaking change in this release is macOS 14 WebKit support removal, which only affects you if you're running WebKit on a 14-year-old OS. Update your browser matrix if that applies.

The @playwright/experimental-ct-svelte package removal is a non-issue unless you were explicitly depending on an experimental package—which you shouldn't be doing in production.

Verdict

Playwright v1.59.0's Screencast API is the feature that makes agentic testing verifiable instead of mysterious. The implementation is clean, the API is intuitive, and the use case is real. It's not a replacement for visual regression tooling, and the storage costs are real, but the observability gains are genuine.

If you're evaluating AI coding agents for test automation, this is the feature that makes the evaluation tractable. You can now watch what the agent did instead of trusting what the agent claims it did. That's not a small thing.

I've shipped test tooling at scale, and the difference between "we have logs" and "we have video evidence" is the difference between debugging in the dark and debugging with a flashlight. The Screencast API gives you the flashlight. Worth exploring in your next sprint.

Porting Anthropic's Skill Creator from Python to TypeScript

Anton Gulin — Thu, 16 Apr 2026 18:11:00 GMT

Why Port It?

Anthropic's skill-creator for Claude Code is excellent. It introduced eval-driven development for AI agent skills — write a skill, test it with evals, optimize the description, benchmark the results. The methodology is proven.

But it has a limitation: it only works with Claude Code, and skill access requires a paid subscription ($20/month minimum). Free tier users can't use it at all.

OpenCode is free and supports 300+ models. I wanted to bring the same methodology to OpenCode users — for free, with no paywall.

High-Level Architecture

The original has this structure:

Anthropic skill-creator/
├── SKILL.md # The skill instructions
├── scripts/
│ ├── run_loop.py # Eval→improve optimization loop
│ ├── improve_description.py # LLM-powered description improvement
│ ├── aggregate_benchmark.py # Benchmark aggregation
│ └── generate_review.py # HTML report generation
└── evals/
└── evals.json # Test query definitions

My version:

opencode-skill-creator/
├── skill-creator/ # The SKILL
│ ├── SKILL.md # Main skill instructions
│ ├── agents/
│ │ ├── grader.md # Assertion evaluation
│ │ ├── analyzer.md # Benchmark analysis
│ │ └── comparator.md # Blind A/B comparison
│ ├── references/
│ │ └── schemas.md # JSON schema definitions
│ └── templates/
│ └── eval-review.html # Eval set review/edit UI
└── plugin/ # The PLUGIN (npm package)
├── package.json # npm package metadata
├── skill-creator.ts # Entry point
└── lib/
├── utils.ts # SKILL.md frontmatter parsing
├── validate.ts # Skill structure validation
├── run-eval.ts # Trigger evaluation
├── improve-description.ts # Description optimization
├── run-loop.ts # Eval→improve loop
├── aggregate.ts # Benchmark aggregation
├── report.ts # HTML report generation
└── review-server.ts # HTTP eval review server

Key difference: the skill provides workflow knowledge, the plugin provides executable tools. The agent orchestrates everything by calling tools during its session.

Decision 1: Scripts → Plugin Tool Calls

Original: Python scripts invoked via CLI

Run the optimization loop

python -m scripts.run_loop --skill-path /path/to/skill --eval-set evals.json

New: Plugin tool calls in OpenCode sessions

skill_optimize_loop with:
evalSetPath: /path/to/evals.json
skillPath: /path/to/skill
maxIterations: 5

Why: OpenCode's plugin architecture lets agents call custom tools directly. No subprocess management, no script execution, no Python environment. The agent calls the tool inline and gets results back in the session.

This is cleaner integration but also more composable. The agent can interleave tool calls with other work — read files, ask the user questions, make decisions — between optimization iterations.

Decision 2: Python → TypeScript

The original requires Python 3.11+ and pyyaml. My version requires nothing beyond Node.js (which OpenCode users already have).

All pipeline components — validation, eval, description improvement, loop runner, aggregation, report generation, review server — are TypeScript modules in the plugin. ~256kB unpacked on npm.

Dependency tree is minimal: the plugin only depends on @opencode-ai/plugin (peer dependency).

Decision 3: Static HTML → HTTP Review Server

Original: Python script generates a static HTML file and opens it in the browser.

generate_review.py --workspace /path/to/workspace

Opens /path/to/workspace/review.html in browser

New: Plugin starts a local HTTP server that serves an interactive eval viewer.

skill_serve_review with:
workspace: /path/to/workspace
skillName: "my-skill"

The HTTP server approach has advantages:

Real-time updates when new eval results come in

Interactive review with save buttons that write feedback back to files

Previous/next navigation between eval cases

Benchmark tab with quantitative metrics

No file management — just open localhost:PORT

The server can also generate static HTML for headless environments:

skill_export_static_review with:
workspace: /path/to/workspace
outputPath: /path/to/report.html

Decision 4: Subagents → Task Tool

Original: Claude Code's built-in subagent concept, where the skill directly spawns sub-agents.

New: OpenCode's Task tool with general and explore subagent types. The SKILL.md instructs the agent to spawn tasks for:

Running eval cases (with-skill and baseline)

Grading assertions against outputs

Analyzing benchmark results

Blind A/B comparison between skill versions

The agent orchestrates these tasks and synthesizes their results.

Decision 5: Staging Outside the Repo

Original: Evals and benchmarks run alongside the skill in the same directory.

New: Draft skills and eval artifacts go to the system temp directory:

/tmp/opencode-skills// # Staged skill
/tmp/opencode-skills/-workspace/ # Eval artifacts

Only the final validated skill gets installed to:

Project: .opencode/skills//

Global: ~/.config/opencode/skills//

This keeps the user's repository clean during skill development. Evals create a lot of artifacts (outputs, timing data, grading results, benchmark files) that you don't want mixed into your project.

Decision 6: Strict Review Workflow

Added a "review workflow guard" that enforces paired comparison data by default:

skill_serve_review and skill_export_static_review require each eval directory to include both with_skill AND baseline (without_skill or old_skill)

If pairs are missing, the tools fail fast with a clear list of what's missing

Override with allowPartial: true only when intentionally reviewing incomplete data

This prevents a common mistake: reviewing eval results without a baseline comparison, which makes it impossible to judge whether the skill actually improved anything.

What I Learned

Skills are software

They need testing, not just writing. The eval-driven approach catches issues you'd never find manually — like a description that triggers on 80% of relevant queries but also fires on 30% of irrelevant ones.

Description optimization matters more than skill content

The description field is the primary triggering mechanism. A well-optimized description on an average skill outperforms a poor description on a perfect skill. This is counterintuitive but matches the data.

Train/test splits prevent overfitting

Same lesson as ML hyperparameter tuning. If you only evaluate on the queries you optimize for, descriptions become overfit. The 60/40 split keeps you honest about generalization.

Human-in-the-loop review is essential

Automation measures triggering accuracy, but humans judge output quality. The visual eval viewer puts outputs side by side so you can see whether the skill produces genuinely useful results, not just correctly-triggered results.

Plugin architecture enables composition

Having eval, benchmarking, and review as separate tool calls (instead of a monolithic script) means the agent can interleave them with other work. It can ask the user a question between iterations, read relevant files during eval, or skip steps the user doesn't need.

Try It

npx opencode-skill-creator install --global

Apache 2.0, free, open source. Works with any of OpenCode's supported models.

GitHub: https://github.com/antongulin/opencode-skill-creator npm: https://www.npmjs.com/package/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global

I Ate My Own Dog Food: How I Benchmarked AI Skills and Proved Eval-Driven Development Works

Anton Gulin — Wed, 15 Apr 2026 18:11:00 GMT

I Ate My Own Dog Food: How I Benchmarked AI Skills and Proved Eval-Driven Development Works

I built a tool to test AI skills. Then I used it on my own project. The benchmarks shocked even me.

As a QA architect, I've spent my career building systems that verify software works correctly. At Apple, we tested everything — every interaction, every edge case, every regression. At CooperVision, I built a Playwright/TypeScript framework from scratch that grew test coverage by 300%.

So when I started working with AI agent skills, I noticed something: nobody was testing them.

You write a SKILL.md file. You try it manually once. Maybe it works for your prompt. You ship it.

There's no automated test suite. No regression testing. No CI pipeline that catches when a description change breaks triggering.

That's a QA problem. I built opencode-skill-creator to solve it.

Then I dogfooded it on a real project. Here's what happened.

The Project: AdLoop Skills for Google Ads

AdLoop is a Google Ads MCP (Model Context Protocol) integration — it connects AI agents to Google Ads and GA4 data through a set of tools.

I created 4 skills for AdLoop using opencode-skill-creator, each handling a different aspect of Google Ads management:

adloop-planning — Keyword research, competition analysis, and budget forecasting

adloop-read — Performance analysis, campaign reporting, and conversion diagnostics

adloop-write — Campaign creation, ad management, keyword bidding, and budget changes (spends real money)

adloop-tracking — GA4 event validation, conversion tracking diagnosis, and code generation

Each skill contains:

A detailed SKILL.md with orchestration patterns, safety rules, and domain-specific best practices

An evals set with test queries (both should-trigger and should-not-trigger)

The full lifecycle: validate → eval → optimize loop → benchmark

The Benchmark: With Skill vs. Without Skill

opencode-skill-creator's benchmark runs each skill through its eval queries in two configurations:

With skill loaded — the AI agent has access to the full SKILL.md with all domain knowledge, safety rules, and orchestration patterns

Without skill — the AI agent only has the bare MCP tool names and descriptions from the schema

The results were striking:

Skill Evals With Skill Without Skill Improvement

adloop-write 8 100% 17% +83 percentage points

adloop-planning 6 100% 21% +79 percentage points

adloop-read 8 100% 27% +73 percentage points

adloop-tracking 6 100% 33% +67 percentage points

100% pass rate across the board — every eval, every expectation — with skills loaded. Without skills, pass rates ranged from 17% to 33%.

But the raw numbers only tell part of the story. Let me show you what actually failed without the skills, because the failures aren't just wrong answers — they're dangerous actions.

The Scariest Failure: adloop-write (17% without skill)

adloop-write manages campaigns, ads, keywords, and budgets. These are operations that spend real money. Without the skill, the AI made these mistakes:

Added BROAD match keywords to MANUAL_CPC campaigns

The #1 cause of wasted ad spend. BROAD match on MANUAL_CPC means Google matches irrelevant queries and drains your budget. The skill explicitly checks bidding strategy before allowing BROAD match. Without it? The AI just adds them.

Set budget above safety caps

The tool has a max_daily_budget of $50. Without the skill, the AI set the budget to $100 — exceeding the cap by 2x. The skill enforces the cap as a guardrail. Without it, no guardrail exists.

Performed irreversible deletions without warning

The skill has a critical safety rule: "Always prefer pause_entity over remove_entity. remove_entity is IRREVERSIBLE." Without the skill, the AI called remove_entity directly — no warning, no confirmation, no pause-as-alternative.

Batched multiple write operations in one call

The skill enforces "one change at a time" — draft, preview, confirm, then next change. Without it, the AI batched pause + sitelink changes together, bypassing review.

This isn't about "better answers." This is about preventing real financial harm.

GDPR Is Not Broken Tracking: adloop-tracking (33% without skill)

The most interesting failure was in adloop-tracking.

A common scenario: a user sees 500 clicks in Google Ads but only 180 sessions in GA4. "Is my tracking broken?"

Without the skill, the AI immediately diagnosed this as a tracking issue and offered to investigate further. It suggested running attribution checks and validating tracking code.

With the skill, the AI recognized this immediately: "A 2.8:1 click-to-session ratio is completely normal with GDPR consent banners. Google Ads counts all clicks regardless of consent. GA4 only records sessions from users who accept analytics cookies. Your tracking is not broken."

This is the #1 false positive in digital marketing analytics. Every marketer who runs EU-targeted ads has seen this panic. The skill prevents hours of investigation and prevents "fixing" something that isn't broken.

Don't Trust Google Blindly: adloop-read (27% without skill)

Google Ads provides automated recommendations. Without the skill, the AI endorsed them at face value:

"Raise budget" — with zero conversions? That's bad advice until tracking works.

"Add BROAD match" — without Smart Bidding? That wastes money.

"More keywords" — with quality scores below 5? The problem is relevance, not volume.

The skill explicitly states: "Google recommendations optimize for Google's revenue, not yours." It cross-references every recommendation against actual conversion data and quality scores before accepting.

The 73% improvement comes from teaching the AI critical thinking, not compliance.

Wrong Country, Wrong Budget: adloop-planning (21% without skill)

Without the skill, the AI defaulted to Germany (the tool's default geo target) when the user asked about US keywords. It didn't group results by competition level. It didn't mention the 5x CPA budget sufficiency rule. It didn't suggest the planning-to-campaign transition workflow.

These aren't edge cases — they're the fundamental decisions that determine whether a campaign succeeds or wastes money.

Why This Matters: Skills Are Safety Guards, Not Nice-to-Haves

The benchmark data tells a clear story: the same AI model, the same tools, the same prompts — the only variable is whether the skill is loaded. And the difference is 67-83 percentage points.

Skills do three things that bare tool access doesn't:

Inject domain expertise

The adloop-write skill knows that BROAD + MANUAL_CPC is the #1 cause of wasted spend. The adloop-tracking skill knows GDPR consent mechanics. The adloop-planning skill knows keyword competition levels and budget rules.

Enforce safety guardrails

Budget caps, irreversible deletion warnings, one-change-at-a-time rules, confirmation prompts before destructive operations. These aren't "context" — they're safety guards that prevent real harm.

Provide orchestration patterns

The skill doesn't just know what each tool does. It knows when to call which tool, in what order, with what validation. It's the difference between a junior dev who knows the API and a senior architect who knows the system.

How to Run Your Own Benchmarks

opencode-skill-creator is free and open source (Apache 2.0). Here's how to benchmark your own skills:

Install
npx opencode-skill-creator install --global

Create a skill with the guided interview
opencode-skill-creator will walk you through it

Run evals with baseline comparison
The tool auto-generates test cases and runs with/without skill

Run the description optimization loop
Train/test split, iterative improvement, variance analysis

Benchmark with visual review
HTML viewer for human QA sign-off

Works with any of OpenCode's 300+ supported models. Zero Python dependency — pure TypeScript.

The Lesson for QA Architects

I've spent my career architecting test systems. At Apple, the standard wasn't "does it work?" — it was "is it perfect?" At CooperVision, I proved that 300% test coverage growth and 50% faster deployments aren't mutually exclusive.

The same discipline applies to AI. Skills are software. They have inputs (prompts), outputs (agent behavior), and a triggering mechanism (the description). They deserve the same testing rigor we apply to any other software.

If you're building AI agent skills and you're not running evals, you're flying blind. Start here:

→ github.com/antongulin/opencode-skill-creator

Skills are software. Software should be tested.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test. Find him at anton.qa or on LinkedIn.

How to Create Custom OpenCode Skills (Step-by-Step Guide)

Anton Gulin — Sun, 12 Apr 2026 18:04:00 GMT

Why Custom Skills Matter

Out-of-the-box AI coding agents are powerful, but they don't know your team's conventions, your deployment process, or your documentation style. Skills let you encode that knowledge so the agent follows your workflows every time.

But creating skills has been guesswork. You write a SKILL.md file, test it manually in a session, maybe tweak the description, and hope it works. There's no feedback loop, no measurement, no way to know if a change actually improved things.

opencode-skill-creator changes this by providing a structured workflow for the full skill lifecycle: create, evaluate, optimize, benchmark, and install.

Prerequisites

OpenCode installed and configured

Node.js 18+ (for the npm package)

5 minutes

Step 1: Install

One command:

npx opencode-skill-creator install --global

This adds the plugin to your global OpenCode config. Restart OpenCode to activate it.

Verify the install:

ls ~/.config/opencode/skills/skill-creator/SKILL.md

Then ask OpenCode: Create a skill that helps with Docker compose files

You should see it use the skill-creator workflow and tools.

Step 2: Describe What You Want

The skill-creator starts with an intake interview. It asks 3-5 targeted questions about what your skill should do:

What should this skill enable OpenCode to do end-to-end?

When should this skill trigger?

What output format and quality bar are expected?

What workflow steps must be preserved vs. where can the agent improvise?

Don't skip this. The interview captures your intent before any code is written. Think of it as shadowing a teammate — you're the domain expert, the agent is the new hire learning your workflow.

Step 3: Review the Skill Draft

Based on your interview, the skill-creator produces a draft SKILL.md with:

Proper YAML frontmatter (name and description)

Markdown instructions for the agent

Optional supporting files (references, agents, templates)

The draft goes to a staging directory (outside your repo) so your project stays clean:

/tmp/opencode-skills/your-skill-name/
├── SKILL.md
├── agents/
├── references/
└── templates/

Review this draft. Make sure the description is accurate (it's the primary triggering mechanism) and the instructions reflect your actual workflow.

Step 4: Generate Eval Test Cases

The skill-creator automatically generates test cases — realistic prompts that an OpenCode user would actually type:

{
"skill_name": "docker-compose",
"evals": [
{
"id": 1,
"prompt": "help me set up a compose file for my Node app with a Postgres database",
"expected_output": "Skill triggers and provides Docker compose guidance",
"should_trigger": true
},
{
"id": 2,
"prompt": "explain how Kubernetes deployments work",
"should_trigger": false
}
]
}

Good eval queries are realistic and specific — not abstract like "help with containers" but concrete like "ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx')..."

Review the eval set. Add or modify test cases that reflect your real usage.

Step 5: Run Evals

The eval system runs each test case twice — once with the skill and once without (baseline). This measures whether the skill actually improves the output.

For each test case:

OpenCode runs with the skill loaded

OpenCode runs without the skill

Both outputs are saved for comparison

Timing data (tokens used, duration) is captured automatically.

Step 6: Review Results Visually

The skill-creator launches an HTML eval viewer:

Call skill_serve_review with:
workspace: /tmp/opencode-skills/your-skill-name-workspace/iteration-1
skillName: "your-skill-name"

The viewer shows:

Outputs tab: Each test case with with-skill and without-skill outputs side by side

Benchmark tab: Quantitative metrics — pass rates, timing, token usage

Feedback fields: Leave comments on each test case

Review the outputs. Give specific feedback on what's working and what's not. Empty feedback means "looks good."

Step 7: Iterate and Improve

Based on your feedback, the skill-creator improves the skill:

Applies your feedback

Reruns all test cases (new iteration)

Launches the reviewer with previous iteration for comparison

You review again

Repeat until you're satisfied or feedback is all empty.

Step 8: Optimize the Description

Even with perfect skill instructions, the skill won't trigger correctly if the description field isn't right. The description is what OpenCode reads to decide whether to load your skill.

The optimization loop:

Generates 20 eval queries (should-trigger and should-not-trigger)

Splits them 60/40 into train/test

Evaluates each query 3 times for statistical reliability

Analyzes failure patterns

LLM proposes improved descriptions

Re-evaluates on both train and test

Selects the best description by test score

Repeats up to 5 iterations

Tell OpenCode:

"Optimize the description of my docker-compose skill"

This takes some time — grab a coffee while it runs.

Step 9: Install the Final Skill

Once you're satisfied with the skill and its description:

Project-level: .opencode/skills/your-skill-name/SKILL.md — available only in this project

Global: ~/.config/opencode/skills/your-skill-name/SKILL.md — available everywhere

Project-level install

cp -r /tmp/opencode-skills/your-skill-name/ .opencode/skills/your-skill-name/

Global install

cp -r /tmp/opencode-skills/your-skill-name/ ~/.config/opencode/skills/your-skill-name/

Only the final validated skill gets installed. All eval artifacts stay in the staging directory.

Real-World Example: Docker Compose Skill

Here's what the full workflow looks like in practice:

Ask OpenCode: "Create a skill that helps with Docker compose files"

Interview: The skill-creator asks about your conventions (multi-service vs. single container, development vs. production, preferred base images)

Draft: Produces a SKILL.md with Docker compose best practices, service configuration patterns, volume mount strategies

Eval: Generates test cases like "my api keeps crashing on startup, can you help me debug my compose file" (should trigger) and "what's the difference between Docker and Podman" (should not trigger)

Review: You look at the outputs, give feedback: "the skill should prioritize security configurations in production compose files"

Iterate: Improved skill draft, better outputs

Optimize: Description goes from "Help with Docker compose files" to something much more specific that triggers reliably

Install: Copy to ~/.config/opencode/skills/docker-compose/

Tips for Great Skills

Be specific in the intake interview: The more context you give, the better the draft

Don't skip evals: They catch triggering issues you'd never find manually

Use realistic test prompts: Write them the way you'd actually type them, typos and all

Iterate at least twice: First drafts are rarely perfect

Optimize the description: It's the #1 factor in whether your skill triggers correctly

Install globally for general skills, project-level for specific ones

Getting Started

npx opencode-skill-creator install --global

Then ask OpenCode to create a skill. That's it.

GitHub: https://github.com/antongulin/opencode-skill-creator

npm: https://www.npmjs.com/package/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global

Page Object Model Playwright (2026): Best TypeScript Guide

Anton Gulin — Sun, 01 Feb 2026 03:38:00 GMT

What is the Page Object Model?

The Page Object Model (POM) is a design pattern that creates an abstraction layer between your tests and your web pages. Instead of writing locators directly in tests, you encapsulate page elements and actions inside reusable classes.

Without POM (hard to maintain):

With POM (scalable):

When the login form changes, you update one file — not 50 tests.

Why Use Page Objects?

Maintainability

Locators live in one place. When UI changes, update once.

Readability

Tests read like user stories: loginPage.login(), dashboard.navigateToSettings().

Reusability

Separation of concerns

Tests focus on behavior. Page objects handle implementation.

Team scalability

New team members understand tests faster.

Project Structure

Here's the structure I use for all my Playwright projects:

Creating a Base Page Class

Start with a base class that all pages extend:

Creating a Login Page Object

Now create a specific page class:

Creating a Dashboard Page Object

Using Page Objects in Tests

Now your tests are clean and readable:

Advanced: Component Objects

For reusable UI components (modals, headers, sidebars), create component classes:

Use it in page objects:

Advanced: Using Playwright Fixtures

For even cleaner tests, integrate page objects with Playwright fixtures:

Now tests are even cleaner:

Best Practices

Use getters for locators

Getters are lazily evaluated. The locator is resolved when accessed, not when the class is instantiated.

Keep actions atomic

Each method should do one logical thing:

Don't add assertions in page objects (usually)

Assertions belong in tests. Page objects encapsulate interactions.

Exception: Verification methods like loginAndExpectDashboard() can include assertions for convenience.

Use role-based locators

More resilient than CSS selectors:

Common Mistakes to Avoid

God page objects

If your page object has 50+ methods, break it into smaller components.

Exposing raw locators everywhere

Encapsulate interactions: loginPage.login() instead of loginPage.submitButton.click().

Not waiting for navigation

After clicks that navigate, wait for the URL or element:

await this.submitButton.click();

await this.page.waitForURL(/.*dashboard/);

Sharing state between tests

Each test should be independent. Don't rely on previous test state.

Folder Structure for Large Projects

For 100+ tests, organize by feature:

Need Architecture Help?

If you're building a test automation framework and want it done right from the start, I can help design the architecture, set up page objects, and train your team on best practices.

Get in touch

How to Migrate from Selenium to Playwright in 2026: Complete Guide

Anton Gulin — Sat, 31 Jan 2026 04:26:00 GMT

Why Migrate from Selenium to Playwright?

I've been using Selenium since 2015. It served me well at Williams-Sonoma, Apple, and countless other companies. But after migrating to Playwright at CooperVision, I won't go back.

Here's how they compare:

Setup time:

Selenium: 30+ minutes

Playwright: 5 minutes

Auto-waiting:

Selenium: Manual (causes flaky tests)

Playwright: Built-in

Cross-browser:

Selenium: Needs separate drivers

Playwright: Bundled

Parallel testing:

Selenium: Requires Grid setup

Playwright: Built-in

Test execution:

Selenium: Slower

Playwright: 40-60% faster

Debugging:

Selenium: Stack traces only

Playwright: Trace viewer with recordings

If your Selenium tests are slow, flaky, or a pain to maintain — migration is worth it.

The Migration Strategy

Don't rewrite everything at once. Follow this incremental approach:

Phase 1: New tests in Playwright

Stop writing new Selenium tests. All new tests go in Playwright.

Phase 2: Migrate critical paths first

Identify your highest-value tests (login, checkout, core features) and migrate those.

Phase 3: Gradual Selenium deprecation

As you touch Selenium tests for maintenance, migrate them. Eventually, the old suite goes to zero.

This approach minimizes risk and lets your team learn Playwright gradually.

Step 1: Set Up Playwright Alongside Selenium

You don't need to remove Selenium to start. Create a parallel Playwright project:

npm init playwright@latest -- --quiet

Your project structure becomes:

Both can run in CI simultaneously during transition.

Step 2: Translate Common Patterns

Here's a side-by-side comparison of common operations:

Navigation:

Selenium (Java):

driver.get("https://example.com");

Playwright (TypeScript):

await page.goto('https://example.com');

Finding elements:

Selenium:

driver.findElement(By.id("username"));

driver.findElement(By.cssSelector(".submit-btn"));

driver.findElement(By.xpath("//button[text()='Submit']"));

Playwright:

page.locator('#username');

page.locator('.submit-btn');

page.getByRole('button', { name: 'Submit' });

Clicking:

Selenium:

driver.findElement(By.id("submit")).click();

Playwright:

await page.locator('#submit').click();

Or better:

await page.getByRole('button', { name: 'Submit' }).click();

Typing:

Selenium:

driver.findElement(By.id("email")).sendKeys("test@example.com");

Playwright:

await page.locator('#email').fill('test@example.com');

Or:

await page.getByLabel('Email').fill('test@example.com');

Waiting:

Selenium (the source of most flakiness):

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("result")));

Playwright (auto-waits by default):

Just interact — Playwright waits automatically

await page.locator('#result').click();

Or explicitly assert visibility:

await expect(page.locator('#result')).toBeVisible();

Assertions:

Selenium (with JUnit):

String title = driver.getTitle();

assertEquals("Expected Title", title);

Playwright:

await expect(page).toHaveTitle('Expected Title');

Screenshots:

Selenium:

File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

FileUtils.copyFile(screenshot, new File("screenshot.png"));

Playwright:

await page.screenshot({ path: 'screenshot.png' });

Step 3: Migrate Page Objects

If you're using Page Object Model in Selenium, the pattern translates directly.

Selenium (Java):

Playwright (TypeScript):

The structure is almost identical. You're just swapping APIs.

Step 4: Replace WebDriver Waits with Auto-Wait

This is where you get the biggest reliability win.

Remove all explicit waits:

In Selenium, you probably have code like:

WebDriverWait wait = new WebDriverWait(driver, 10);

wait.until(ExpectedConditions.elementToBeClickable(By.id("button")));

driver.findElement(By.id("button")).click();

In Playwright, just:

await page.locator('#button').click();

Playwright automatically waits for:

Element to be attached to DOM

Element to be visible

Element to be enabled

Element to be stable (not animating)

This single change eliminates most flaky tests.

Step 5: Set Up CI/CD

Playwright's CI setup is simpler than Selenium Grid.

GitHub Actions example:

No Selenium Grid. No browser driver management. Just install and run.

Common Migration Pitfalls

Trying to rewrite everything at once

This is the number one cause of failed migrations. Use the incremental approach.

Copying Selenium's explicit waits

Don't translate WebDriverWait to waitForSelector. Trust Playwright's auto-waiting.

Keeping fragile locators

Migration is a chance to improve. Replace XPath and positional CSS with role-based locators.

Not training the team

Playwright has different patterns. Invest in a training session or pair programming during the first weeks.

Migration Checklist

Before declaring migration complete:

All new tests written in Playwright

Critical user journeys migrated

CI/CD running Playwright tests

Team trained on Playwright patterns

Old Selenium tests deleted (don't leave dead code)

Trace viewer configured for failure debugging

How Long Does Migration Take?

Based on my experience:

Small suite (under 50 tests): 1-2 weeks

Medium suite (50-200 tests): 1-2 months

Large suite (200+ tests): 3-6 months

The biggest factor isn't the number of tests — it's how tangled your Selenium architecture is. Clean Page Objects migrate fast. Spaghetti code requires refactoring.

The Results Are Worth It

At CooperVision, after completing migration:

Test execution time: 40% faster

Flaky test rate: Reduced by 80%

CI pipeline reliability: 98%+ pass rate

Developer experience: "I actually like writing tests now"

That last point matters more than any metric. When tests are pleasant to write, people write more tests.

Need Help with Your Migration?

If you're planning a Selenium to Playwright migration and want guidance from someone who's done it multiple times, I can help. I offer consulting packages that include:

Migration strategy and roadmap

Team training on Playwright patterns

Architecture review of existing tests

Hands-on migration for critical paths

Let's talk

How to Fix Flaky Tests in Playwright: 10 Battle-Tested Strategies

Anton Gulin — Fri, 30 Jan 2026 05:16:00 GMT

The Cost of Flaky Tests

A flaky test is one that sometimes passes and sometimes fails without any code changes. They seem harmless at first — just re-run the pipeline, right?

Wrong. Flaky tests are silent killers:

Eroded trust — Teams stop believing test results

Wasted time — Engineers debug tests instead of building features

Slower releases — "Just re-run it" becomes the norm

Hidden bugs — Real failures get dismissed as "flaky"

At one company, I inherited a test suite with a 68% pass rate. Not because the application was broken — but because the tests were. Here's how I fixed it.

Strategy 1: Use Role-Based Locators

The problem: CSS selectors and XPath break when developers change class names or restructure HTML.

The fix: Use Playwright's role-based locators:

Bad:

page.locator('#submit-btn')

page.locator('.form-container > button:nth-child(2)')

Good:

page.getByRole('button', { name: 'Submit' })

page.getByLabel('Email address')

page.getByText('Welcome back')

Role-based locators are more resilient because they target what users see, not implementation details.

Strategy 2: Never Use Hard-Coded Waits

The problem: waitForTimeout() is the number one cause of flaky tests.

await page.waitForTimeout(5000); // ❌ Please don't do this

Why it fails: 5 seconds might be enough on your machine but not in CI. Or it might be way too long, slowing tests unnecessarily.

The fix: Wait for specific conditions:

await page.waitForLoadState('networkidle');

await expect(page.getByRole('button')).toBeEnabled();

await expect(page.getByText('Success')).toBeVisible();

Playwright's auto-waiting handles most cases automatically. Trust it.

Strategy 3: Isolate Test State

The problem: Tests depend on state from previous tests.

test('login', ...); // Creates session

test('add to cart', ...); // Expects logged-in state

If the login test fails, the cart test also fails — but not because of a cart bug.

The fix: Each test should set up its own state:

Or use Playwright's storage state to share authentication without dependencies:

await page.context().storageState({ path: 'auth.json' });

Strategy 4: Handle Loading States Explicitly

The problem: Clicking a button that's still loading, or reading text before it's rendered.

The fix: Wait for loading indicators to disappear:

Or wait for the element to be in a specific state:

Strategy 5: Use data-testid for Dynamic Content

The problem: Elements generated dynamically have unpredictable locators.

The fix: Add data-testid attributes for testing:

In your application code:

In your test:

await page.getByTestId('checkout-button').click();

This creates a contract between frontend and tests that survives refactoring.

Strategy 6: Retry Failed Assertions (Not Whole Tests)

The problem: A test fails once and you re-run the entire suite.

The fix: Use Playwright's built-in expect retries:

Assertions like toBeVisible() and toHaveText() will automatically retry until timeout — no manual retries needed.

Strategy 7: Handle Network Variability

The problem: API calls take longer in CI than locally.

The fix: Wait for network responses explicitly:

Or use networkidle for simpler cases:

await page.goto('/dashboard', { waitUntil: 'networkidle' });

Strategy 8: Run Tests in Parallel Correctly

The problem: Tests interfere with each other when running in parallel.

Test A creates user "testuser@example.com"

Test B also creates user "testuser@example.com"

One fails due to duplicate email.

The fix: Use unique data per test:

Or isolate tests in separate browser contexts (Playwright does this by default).

Strategy 9: Use Trace Viewer for Debugging

The problem: You can't see what happened when a test failed in CI.

The fix: Enable traces on failure:

Now, when a test fails and retries, Playwright captures:

Screenshots at every step

DOM snapshots

Network requests

Console logs

Open traces with:

npx playwright show-trace trace.zip

This is the single best debugging tool for flaky tests.

Strategy 10: Set Realistic Timeouts

The problem: Default timeouts are too short for slow environments.

The fix: Configure appropriate timeouts based on your CI environment:

Don't make them too long — slow failures are frustrating. Find the right balance for your infrastructure.

Bonus: The Flaky Test Triage Process

When you encounter a flaky test, follow this process:

Reproduce locally

Run the test 10 times:

npx playwright test tests/checkout.spec.ts --repeat-each=10

If it passes every time locally but fails in CI, it's likely a timing or environment issue.

Check the trace

Open the trace file from CI and look for:

Slow network requests

Elements not visible when clicked

Unexpected modal or popup

Identify the root cause

Common causes:

Hard-coded wait times

Race conditions

Shared test state

Unstable locators

Fix or delete

If you can't fix it after 30 minutes, delete it. A flaky test is worse than no test. You can always rewrite it properly later.

My Flaky Test Scorecard

After applying these strategies at CooperVision, we went from 68% pass rate to 98%+ in three months.

Pass rate:

Before: 68%

After: 98.5%

Average test time:

Before: 4.2 minutes

After: 1.8 minutes

"Re-run pipeline" requests:

Before: Daily

After: Rare

Team trust in automation:

Before: Low

After: High

The biggest win wasn't technical — it was cultural. When tests are reliable, developers actually care about failures.

Need Help with Your Flaky Tests?

If your test suite is unreliable and slowing down your team, I can help. I've fixed test suites at Fortune 500 companies and can usually identify the main issues in just a few hours.

Book a consultation

Playwright Tutorial for Beginners: Your First Test in 10 Minutes

Anton Gulin — Thu, 29 Jan 2026 06:02:00 GMT

What You'll Learn

By the end of this tutorial, you'll have:

Playwright installed on your machine

Written and run your first automated test

Understood the basic Playwright concepts

A foundation to build real-world test suites

Total time: about 10 minutes. Let's go.

Prerequisites

You need Node.js 18+ installed. Check your version:

node --version

If you need to install Node.js, download it from nodejs.org.

That's it. No other dependencies required.

Step 1: Create Your Project

Open your terminal and create a new folder:

mkdir playwright-tutorial

cd playwright-tutorial

Step 2: Install Playwright

Run the Playwright installer:

npm init playwright@latest

You'll see some prompts. Accept the defaults:

TypeScript or JavaScript? → TypeScript (recommended)

Where to put tests? → tests

Add GitHub Actions? → Yes (optional, but useful later)

Install browsers? → Yes

This installs Playwright, downloads browser binaries, and creates a basic project structure.

Step 3: Explore the Project Structure

After installation, you'll see:

tests/ — Your test files go here

playwright.config.ts — Configuration file

tests/example.spec.ts — A sample test (we'll replace this)

Step 4: Write Your First Test

Delete the example test and create a new file called tests/first.spec.ts:

What this does:

test() — Defines a test with a name

page.goto() — Navigates to a URL

expect().toHaveTitle() — Asserts the page title contains "Playwright"

Step 5: Run Your Test

In your terminal:

npx playwright test

You should see:

Congratulations! You just ran your first Playwright test.

Step 6: See the Test Report

Playwright generates beautiful HTML reports. View it:

npx playwright show-report

This opens a browser with your test results, including timing, screenshots, and traces.

Step 7: Run Tests in UI Mode (Game Changer)

Playwright's UI mode lets you watch tests run in real-time:

npx playwright test --ui

This opens a visual interface where you can:

Watch tests execute step-by-step

Inspect DOM at any point

Debug failures visually

Re-run individual tests

This is my favorite Playwright feature for development.

Step 8: Add More Assertions

Let's expand our test to click a link and verify navigation:

New concepts:

getByRole() — Finds elements by their accessibility role (best practice)

click() — Clicks an element

toHaveURL() — Asserts the current URL matches a pattern

toBeVisible() — Asserts an element is visible on page

Step 9: Use the Code Generator

Playwright can generate test code by recording your actions:

npx playwright codegen playwright.dev

This opens a browser. Click around, and Playwright writes code for you in real-time. Copy the generated code into your test file.

This is extremely useful for learning locators and speeding up test creation.

Understanding Key Concepts

Locators:

Playwright finds elements using locators. Best practices:

getByRole() — By accessibility role (button, link, heading)

getByText() — By visible text

getByLabel() — By form label

getByTestId() — By data-testid attribute

Avoid CSS selectors and XPath when possible — role-based locators are more resilient.

Auto-waiting:

Playwright automatically waits for elements to be ready before interacting. No more:

await page.waitForSelector('.button'); // Not needed!

Just write:

await page.click('.button'); // Playwright handles waiting

This dramatically reduces flaky tests.

Next Steps

Now that you have the basics:

Add more tests — Start testing your own application

Learn Page Object Model — Organize tests for maintainability

Set up CI/CD — Run tests on every pull request

Explore API testing — Playwright also tests REST APIs

Common Beginner Mistakes to Avoid

Using sleep/delay:

Bad:

await page.waitForTimeout(5000);

Good: Let Playwright's auto-waiting handle it, or wait for specific conditions.

Fragile locators:

Bad:

page.locator('#app > div:nth-child(3) > button')

Good:

page.getByRole('button', { name: 'Submit' })

Not using test isolation:

Each test should be independent. Don't rely on state from previous tests.

Resources

Official docs: playwright.dev/docs/intro

Test generator:

npx playwright codegen [url]

UI mode:

npx playwright test --ui

Report viewer:

npx playwright show-report

Need Hands-On Help?

If you're building a test automation framework for your team and want guidance from someone who's done it at Apple and Fortune 500 companies, I offer consulting and training.

Learn more about working with me

Playwright vs Cypress vs Selenium in 2026: The Definitive Comparison

Anton Gulin — Wed, 28 Jan 2026 05:01:00 GMT

Why This Comparison Matters in 2026

After 10+ years of building test automation frameworks at Apple, Fortune 500 companies, and fast-moving startups, I've worked with all three tools extensively. This isn't a theoretical comparison — it's based on real production experience.

The test automation landscape has shifted dramatically. Playwright has matured into a serious contender, Cypress has expanded beyond its original scope, and Selenium remains the enterprise backbone. But which one should you actually choose?

Let me break it down.

Quick Comparison

Playwright:

Language Support: TypeScript, JavaScript, Python, Java, C#

Browser Support: Chromium, Firefox, WebKit

Speed: Very Fast

Learning Curve: Moderate

Price: Free

Cypress:

Language Support: JavaScript/TypeScript only

Browser Support: Chromium, Firefox, Edge, WebKit

Speed: Fast

Learning Curve: Easy

Price: Free (paid for some features)

Selenium:

Language Support: All major languages

Browser Support: All browsers

Speed: Moderate

Learning Curve: Steeper

Price: Free

Playwright: The New Standard

Playwright has become my go-to framework for new projects. Built by Microsoft (the same team that originally created Puppeteer), it addresses almost every pain point I've encountered with other tools.

What Playwright does best:

True cross-browser testing — Chrome, Firefox, Safari (WebKit) from a single API

Auto-waiting — No more flaky

waitFor

statements

Built-in parallelization — Shards tests automatically across workers

Native TypeScript support — First-class, not bolted on

Trace viewer — Debug failures with recordings, network logs, and DOM snapshots

API testing included — Test REST endpoints with the same framework

When to choose Playwright:

Starting a new automation framework from scratch

Need cross-browser testing (including Safari)

Team knows TypeScript or willing to learn

Want modern DX (developer experience

Cypress: Still Strong, But With Caveats

Cypress pioneered the "developer-friendly testing" movement. Its interactive test runner and real-time reloading changed how developers think about testing.

What Cypress does best:

Developer experience — The interactive UI is unmatched for debugging

Easy setup — npm install and you're running tests

Excellent documentation — Some of the best in the industry

Component testing — Strong support for React, Vue, Angular

The catches:

No native Safari support — WebKit support is experimental

Parallel testing is paid — Cypress Cloud subscription required

Origin limitations — Testing across multiple domains can be tricky

Slower execution — Runs inside the browser, not alongside it

When to choose Cypress:

Small team, primarily Chrome users

Heavy component testing needs

Budget for Cypress Cloud

JavaScript-only team

Selenium: The Enterprise Workhorse

Selenium has been around for almost 20 years. It's not the flashiest choice, but it's battle-tested across every major enterprise on the planet.

What Selenium does best:

Universal language support — Java, Python, C#, Ruby, JavaScript

Massive ecosystem — Extensive tooling, plugins, and community

Enterprise trust — Every company has Selenium experience

Any browser — True universal browser support

Mobile via Appium — Extends to native mobile testing

The challenges:

No auto-waiting — You manage waits manually (flakiness source)

More boilerplate — More code to achieve the same result

Slower test execution — WebDriver protocol overhead

Grid complexity — Parallel execution requires infrastructure

When to choose Selenium:

Large enterprise with existing Selenium investment

Need languages Playwright doesn't support

Extending to mobile testing with Appium

Team has deep Selenium expertise

Real-World Performance Comparison

I ran the same test suite (50 tests, e-commerce checkout flow) across all three frameworks. Here's what I found:

Setup time:

Playwright: 5 minutes

Cypress: 5 minutes

Selenium: 15 minutes

Test execution (50 tests):

Playwright: 45 seconds

Cypress: 72 seconds

Selenium: 98 seconds

Flaky test rate:

Playwright: 2%

Cypress: 8%

Selenium: 15%

CI/CD complexity:

Playwright: Low

Cypress: Medium

Selenium: High

Playwright's auto-waiting and parallel execution give it a significant edge in both speed and reliability.

My Recommendation for 2026

For new projects: Playwright. The developer experience, cross-browser support, and performance make it the best choice going forward.

For existing Cypress projects: Stay with Cypress if it's working. Migrate only if you're hitting limitations (Safari testing, parallel costs).

For existing Selenium projects: Consider a gradual migration to Playwright. The migration path is cleaner than you might expect — many patterns transfer directly.

Making the Final Decision

Ask yourself these questions:

Do you need Safari/WebKit testing? → Playwright

Is your team JavaScript-only with limited testing experience? → Cypress

Do you have significant Selenium investment and enterprise constraints? → Selenium (with migration roadmap)

Starting fresh with no legacy code? → Playwright, every time

The frameworks are all capable of building production-quality test automation. The right choice depends on your team, your constraints, and where you're headed — not where you've been.

Need Help Choosing or Migrating?

If you're evaluating frameworks or considering a migration from Selenium or Cypress to Playwright, I can help. I've led migrations at Fortune 500 companies and can have your first Playwright tests running in a week.

Get in touch

Anton Gulin — Test Automation Blog

Your Regression Suite Is a Museum: 5 Questions That Decide Delete vs Keep

Your Regression Suite Is a Museum: 5 Questions That Decide Delete vs Keep

Question 1: When did this test last fail for a real reason?

Question 2: What user risk does it guard?

Question 3: Would anyone notice if it vanished?

Question 4: Does it check results or steps?

Question 5: Can it fail at all?

What you get back

How to Review AI-Generated Tests: Seven Checks Before You Keep Them

Post

1. Name the user risk

2. Break the product

3. Check the final result

4. Change the data

5. Read the failure

6. Run it twice

7. Make the release decision

A five-minute review example

Use the card during review

Test Retries Hide Real Bugs: When a Rerun Helps and When It Lies

Post

What the rerun actually changed

The race that a rerun hides

Playwright already tells you the truth

The setup that keeps retries useful

The rule

Reuse One Page Object Method for Success and Failure Cases

Post

The duplicate everyone has

Why the boolean flag makes it worse

Copy the shape Playwright already uses

Notice what the option does not say

When two methods really are right

The rule, in three lines

Should Page Objects Assert? Where Test Assertions Belong

Post

The two kinds of checks

The rule

Why hidden business asserts hurt

Where it gets honestly debatable

The migration path

The takeaway

The Modern Page Object Model: Less Shared Code, Easier Changes

Sign 1: Every page object extends BasePage

Sign 2: Wait methods everywhere

Sign 3: Locators buried inside methods

Sign 4: Tests build their own page objects

Sign 5: Page objects that assert

The migration path (no rewrite required)

Do AI agents change any of this?

The pattern in one line

Your AI Model Is a Dependency: Pin It, Keep a Fallback, Re-Verify

What "the model is a dependency" actually means

Rule 1 — Pin the version

Rule 2 — Keep a fallback you have already validated

Rule 3 — Re-run the same suite on both

What this cost the two kinds of teams

The checklist

How to Implement AI in QA (2026): A Practical Framework

How to Implement AI in QA (2026): A Practical Framework

Key takeaways

What "AI in QA" actually means

The one rule that keeps it safe

Where AI helps in QA (the 4 good jobs)

Where AI lies in QA (the 4 traps)

A working example: an AI agent with an oracle it cannot move

The 3 tests every AI feature needs

Where to start on Monday

How to Test Passkey (WebAuthn) Login in Playwright (2026)

How to Test Passkey (WebAuthn) Login in Playwright (2026)

Key takeaways

What is a passkey, in plain words

Why nobody tested this flow

What Playwright 1.61 added

How to test passkey login — a working example

Re-using a passkey across tests

The honest part: what this does not test

Why this matters for your CI pipeline

Playwright Best Practices: 10 Rules AI Agents Get Wrong (2026)

Sign 1: Every page object extends `BasePage`

8. Keep `if` and `try` out of your tests