43AI in Testing · MCP · Agents

43.1 Where AI helps in SDET work (honestly)

43.2 AI-assisted test authoring

The workflow

1. Open the codebase + spec in Cursor / Copilot / Claude Code.
2. Ask: "Write a Playwright TS test for: a user signs in, navigates to settings, changes their email, verifies a confirmation email appears in their inbox."
3. Review carefully — AI may invent selectors. Run, fix, refactor.
4. Ask AI to refactor into a Page Object once stable.

// AI first draft — verify and refine
import { test, expect } from '@playwright/test';

test('user updates email and sees confirmation', async ({ page, request }) => {
  // ... AI's attempt ...
});
// Often locators need cleanup, fixtures need swapping in. Don't merge AI output as-is.

43.3 AI-assisted debugging

Workflow: a test fails on CI, you download the trace, paste a summary into your AI assistant. "This test fails on CI but passes locally. Trace shows action .click() on getByRole('button', { name: 'Submit' }) times out. Network shows POST /submit returns 502. What are common causes?" The AI returns a checklist (CORS, timing, env diff, upstream flakiness).

43.4 Self-healing locators

See Module 26 §26.6 for the architecture. In practice today this is offered by tools like Healenium, Functionize, Mabl. Playwright doesn't ship one out of the box, but you can build one over the captured Locator + DOM snapshot from the trace.

43.5 LLM-as-judge

For features whose output isn't deterministic (chat, summarisation, search), build a "judge" model that scores the output. See Module 10 §10.6.

43.6 Model Context Protocol (MCP)

MCP is an open protocol (introduced by Anthropic in late 2024) for connecting LLMs to tools and context sources in a standard way. Think of it as USB-C for AI: instead of every app inventing its own way to expose tools (Slack, GitHub, your DB), a tool exposes itself as an MCP server, and any MCP-compatible client (Claude Desktop, Cursor, custom agents) can use it.

The three roles

What a server exposes

• Tools — functions the LLM can call (browse, click, fill, take screenshot)
• Resources — data the LLM can read (file content, DB rows, API responses)
• Prompts — reusable prompt templates the user can trigger

43.7 Playwright MCP server

Microsoft ships @playwright/mcp — an MCP server that exposes Playwright operations (open URL, click, screenshot, get accessibility tree) as tools. An LLM with this connection can "drive a browser":

npx @playwright/mcp@latest
# Or wire into Claude Desktop config:
{
  "mcpServers": {
    "playwright": { "command": "npx", "args": ["@playwright/mcp@latest"] }
  }
}

Now in Claude Desktop you can say: "Go to example.com/login, sign in as qa@test.com, take a screenshot of the dashboard." Claude calls the MCP server's tools in sequence and reports back.

Use cases for SDETs

• Exploratory testing — "Explore this app for 10 minutes and report any anomalies"
• Test case authoring — Claude drives the app once, then writes the Playwright test based on what it did
• Bug reproduction — Paste a bug report; Claude tries the steps and reports outcome
• Visual review — Claude grabs screenshots across pages for design review

43.8 Playwright "Agents" (Test-AI workflows)

The phrase "Playwright Agent" is used in two ways:

1. An LLM connected to the Playwright MCP server, performing browser tasks autonomously. Discussed above.
2. The Playwright codegen-like "Agent" experience in newer tooling — natural-language prompts that produce Playwright code, then run it, observe failures, retry. Tools like TestMu AI (LambdaTest), Auto Playwright, ZeroStep.

// Auto Playwright — let an LLM decide the next action
import { test, expect } from '@playwright/test';
import { auto } from 'auto-playwright';

test('semantic-driven test', async ({ page }) => {
  await page.goto('https://shop.example.com');
  await auto('search for headphones', { page, test });
  await auto('add the first result to the cart', { page, test });
  await auto('go to checkout', { page, test });
  await expect(page).toHaveURL(/checkout/);
});

43.9 Testing AI features in YOUR product

Covered in Module 10 §10.6. Quick recap:

1. Mock the model endpoint in CI for deterministic UI tests.
2. Property-based assertions on real output (length, format, contains key terms, no PII).
3. Golden prompt set + LLM judge for quality regressions.
4. Prompt-injection adversarial tests (direct + indirect via retrieved docs).
5. Refusal tests, bias tests, drift detection on schedule.

43.10 What does the future of SDET look like?

• Test maintenance becomes LLM-assisted — "broken locator? AI suggests fix, you click approve"
• Exploratory testing partially automated — "agent, find UI bugs in this build" returns a triaged list
• Bug reports become reproducible Playwright scripts automatically
• Non-deterministic features routinely tested with LLM-judge pipelines
• Your job shifts upward: from writing tests to designing test strategy, gating quality, owning observability