36SDET System Design

36.1 How interviewers grade this round

SDET system design at 5+ years experience asks: "Design a Y at scale for an org with N tests / N devs / N envs." They look for:

• You ask clarifying questions before drawing — scale, constraints, who uses it
• You separate functional requirements (what it does) from non-functional (latency, throughput, availability)
• You sketch components and APIs explicitly
• You discuss trade-offs (queue vs sync; SQL vs key-value; push vs poll)
• You estimate capacity (back-of-envelope numbers)
• You spot bottlenecks and propose mitigations

36.2 Common SDET system-design prompts

1. "Design a test execution platform for 10K tests across 50 microservices, running on every PR within 10 minutes."
2. "Design a test data management service that gives each test isolated, realistic data on demand."
3. "Design a flaky-test detection and quarantine pipeline."
4. "Design a self-healing locator system."
5. "Design a unified test results dashboard across teams."

36.3 Worked example — distributed test execution platform

(1) Clarify

• How many tests total? (10,000 UI + 50,000 API)
• How many PRs per day? (500)
• SLA for feedback? (PR feedback < 10 min)
• Browsers / OSes? (Chromium / Firefox / WebKit on Linux)
• Cloud or self-hosted? (AWS, can use spot instances)
• Who runs it? (Platform team, ~5 SDETs)

(2) Functional requirements

• Trigger on PR / push / cron / manual
• Select which tests to run for a given diff (test selection)
• Distribute across N workers (smart sharding)
• Collect results, traces, screenshots
• Merge into one unified report
• Gate PR merge on result
• Detect and quarantine flakes automatically

(3) Non-functional

• p95 feedback time < 10 min
• 5000 tests/PR × 500 PRs/day = 2.5M test runs/day. Burst capacity at peak hours.
• Availability: 99.9% (downtime blocks all merges)
• Cost: target < $X per PR run

(4) Architecture sketch

(5) Key design decisions

(6) Capacity estimate

• 2.5M tests/day. Peak ~10× average over 1 hour = 700 tests/sec.
• Each test = ~30 s in browser. So ~21,000 concurrent worker slots at peak.
• ~700 worker pods × 30 slots/pod (40 GB nodes). Manageable on EKS spot fleet.
• Queue throughput: SQS handles 700/sec trivially; Kafka if we need ordering / replay.
• Artifact store: ~10 MB trace/video per test × 2.5M = 25 TB/day. Set 14-day retention.

(7) Trade-offs to discuss

• SQS vs Kafka — SQS simpler, no ordering. Kafka if we need replay or strict ordering.
• Diff-based selection vs run-all — selection risks missing affected tests; combine with nightly full regression.
• Spot vs on-demand — spot cheaper, can be reclaimed mid-test. Build retry/redispatch.
• Synchronous result vs polling — GitHub status check polls; webhooks on result table can push updates.

36.4 Smart test selection

Run only tests likely affected by the diff. Several approaches:

• Path-based — file change in /checkout/ triggers tests tagged @checkout. Static mapping, simple, brittle.
• Coverage-based — at last green run, capture coverage per test. Map line changes to tests. Accurate but heavy infrastructure.
• ML-based — train on historical PR → broken-test pairs. Probabilistic. Powerful but needs data.
• Test impact analysis (TIA) — combination above + safety net of full regression nightly.

36.5 Test Data Service

Goal: every test gets isolated, realistic data, in milliseconds. Options:

36.6 Self-healing locators (AI-assisted)

When a locator breaks because the DOM changed, the system tries to find the best alternative automatically. Approach:

1. Capture a rich snapshot per locator on green runs (role, name, neighbours, text, position, attributes).
2. On failure, query the current DOM for candidates that score highly against the snapshot.
3. Suggest top match in the report; allow auto-apply if confidence > 0.9.
4. Surface healed locators in PR for human review — never silently change tests.

36.7 Reporting analytics

• Time-series of pass rate, retry rate, mean run time per test
• Top flaky tests this week
• Top slowest tests
• Coverage trend
• PR feedback latency histogram
• Cost per PR

36.8 Common follow-up questions

• "What if a worker crashes mid-test?" — retry on a different worker, mark the original as failed/retried.
• "How do you handle test data tied to a specific tenant?" — tenant pool with leases; expire stale leases.
• "How do you keep auth tokens fresh across thousands of tests?" — central token service issues short-lived tokens; cache per worker.
• "How do you handle a flaky third-party API in tests?" — proxy with HAR cache; replay if upstream slow.
• "How do you debug a single failing test in this system?" — trace stored per test, URL surfaced in PR comment.

36.4 Implementation Patterns

Test Execution Queue — Python + Redis

import redis
import json
import uuid
from dataclasses import dataclass, field
from typing import List

r = redis.Redis(host="localhost", port=6379)

@dataclass
class TestJob:
    job_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    spec_file: str = ""
    browser: str = "chromium"
    shard: int = 1
    total_shards: int = 1
    env: str = "staging"

def enqueue_test_run(specs: List[str], browsers: List[str], total_shards: int = 4) -> str:
    run_id = str(uuid.uuid4())
    jobs = []
    for spec in specs:
        for browser in browsers:
            for shard in range(1, total_shards + 1):
                job = TestJob(spec_file=spec, browser=browser,
                              shard=shard, total_shards=total_shards)
                jobs.append(job)
    pipe = r.pipeline()
    for job in jobs:
        pipe.lpush("test:queue", json.dumps(job.__dict__))
    pipe.set(f"run:{run_id}:total", len(jobs))
    pipe.set(f"run:{run_id}:pending", len(jobs))
    pipe.execute()
    return run_id

def worker_loop():
    while True:
        raw = r.brpop("test:queue", timeout=30)
        if raw is None:
            break
        job = TestJob(**json.loads(raw[1]))
        run_playwright_shard(job)

def run_playwright_shard(job: TestJob):
    import subprocess
    cmd = [
        "npx", "playwright", "test", job.spec_file,
        "--project", job.browser,
        f"--shard={job.shard}/{job.total_shards}",
        "--reporter=json"
    ]
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
    r.lpush(f"run:results", json.dumps({
        "job_id": job.job_id,
        "spec": job.spec_file,
        "browser": job.browser,
        "exit_code": result.returncode,
        "stdout": result.stdout[-2000:],
    }))

Test Data Service — FastAPI

from fastapi import FastAPI, HTTPException
import psycopg2, uuid, json
from contextlib import contextmanager

app = FastAPI()
DB_DSN = "postgresql://user:pass@localhost/testdb"

@contextmanager
def get_conn():
    conn = psycopg2.connect(DB_DSN)
    try:
        yield conn
        conn.commit()
    finally:
        conn.close()

@app.post("/v1/test-data/user")
def create_test_user(template: str = "standard"):
    uid = str(uuid.uuid4())[:8]
    user = {
        "email": f"test_{uid}@example.com",
        "password": "TestPass123!",
        "role": "user" if template == "standard" else "admin",
    }
    with get_conn() as conn:
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO users(email,password,role,test_session_id) VALUES(%s,%s,%s,%s) RETURNING id",
            (user["email"], user["password"], user["role"], uid)
        )
        user["id"] = cur.fetchone()[0]
    return user

@app.delete("/v1/test-data/cleanup/{session_id}")
def cleanup_session(session_id: str):
    with get_conn() as conn:
        cur = conn.cursor()
        cur.execute("DELETE FROM users WHERE test_session_id = %s", (session_id,))
        deleted = cur.rowcount
    return {"deleted": deleted}

Self-Healing Locator — TypeScript

import { Page, Locator } from "@playwright/test";

interface LocatorStrategy {
  strategy: string;
  value: string;
  score: number;
}

export async function selfHealingLocator(
  page: Page,
  strategies: LocatorStrategy[]
): Promise<Locator> {
  const sorted = strategies.sort((a, b) => b.score - a.score);
  for (const s of sorted) {
    let loc: Locator;
    if (s.strategy === "data-testid") loc = page.locator(`[data-testid="${s.value}"]`);
    else if (s.strategy === "role") loc = page.getByRole(s.value as any);
    else if (s.strategy === "text") loc = page.getByText(s.value);
    else loc = page.locator(s.value);

    if ((await loc.count()) > 0) {
      console.log(`Healed via ${s.strategy}: ${s.value}`);
      return loc;
    }
  }
  throw new Error(`All locator strategies failed: ${JSON.stringify(strategies)}`);
}

// Usage in test
// const btn = await selfHealingLocator(page, [
//   { strategy: "data-testid", value: "submit-btn", score: 100 },
//   { strategy: "role",        value: "button",     score:  80 },
//   { strategy: "text",        value: "Submit",      score:  60 },
// ]);