55Chaos Engineering

55.1 The discipline

Chaos Engineering = deliberately injecting failures into a running system to discover weaknesses before users do. Born at Netflix (Chaos Monkey, 2010s).

Five principles

1. Define a steady-state hypothesis — what measurable behaviour means "system is fine"? Latency p95, error rate, transactions/sec.
2. Vary real-world events — kill a node, drop network packets, fill disk, throttle CPU.
3. Run in production (eventually) — staging won't surface every issue. Start in pre-prod, graduate as confidence grows.
4. Automate experiments to run continuously
5. Minimise blast radius — abort if steady state breaks, start small.

55.2 Failure modes to inject

55.3 Tools

55.4 Sample experiment (Chaos Mesh)

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: latency-payments
spec:
  action: delay
  mode: all
  selector:
    namespaces: [payments]
    labelSelectors: { app: charge-service }
  delay:
    latency: '500ms'
    jitter: '100ms'
    correlation: '50'
  duration: '5m'

Apply with kubectl apply -f experiment.yaml. Watch dashboards. Recover with kubectl delete.

55.5 Designing a chaos experiment

1. Pick one failure (e.g. "downstream service slows by 500 ms").
2. State the steady-state hypothesis ("checkout p95 stays < 3 s; error rate < 1%").
3. Limit blast radius (one canary instance, off-peak window).
4. Define abort conditions ("if error rate > 5% for 2 min, halt").
5. Run, observe dashboards, capture findings.
6. If weakness found → file bug → fix → re-run.

55.6 Game days

Scheduled session where engineers run a series of chaos experiments live, with the on-call team observing and responding. Practices incident response, exposes gaps in runbooks/dashboards. Quarterly is a common cadence.

55.7 SDET role

• Author the chaos experiments + assertions
• Define the steady-state metrics (often borrowed from SLOs)
• Integrate chaos into nightly / weekly schedule (not every PR)
• Verify retry/circuit-breaker code paths trigger correctly
• Update runbooks based on findings

Chaos Engineering with Chaos Toolkit

pip install chaostoolkit chaostoolkit-kubernetes chaostoolkit-aws

# Run chaos experiment
chaos run experiment.json

# Validate experiment file
chaos validate experiment.json

# View experiment report
chaos report --export-format=pdf chaos-journal.json report.pdf

Chaos Experiment Definition

{
  "version": "1.0.0",
  "title": "API Service Survives Database Outage",
  "description": "Verify the API returns 503 gracefully when database is unavailable",
  "tags": ["database", "resilience", "api"],
  "steady-state-hypothesis": {
    "title": "API returns 200 OK",
    "probes": [
      {
        "name": "api-health-check",
        "type": "probe",
        "tolerance": 200,
        "provider": {
          "type": "http",
          "url": "http://localhost:8080/health",
          "timeout": 3
        }
      }
    ]
  },
  "method": [
    {
      "type": "action",
      "name": "stop-database-container",
      "provider": {
        "type": "process",
        "path": "docker",
        "arguments": "stop postgres-db"
      }
    },
    {
      "type": "probe",
      "name": "api-returns-503-gracefully",
      "tolerance": 503,
      "provider": {
        "type": "http",
        "url": "http://localhost:8080/api/users",
        "timeout": 5
      }
    }
  ],
  "rollbacks": [
    {
      "type": "action",
      "name": "restart-database",
      "provider": {
        "type": "process",
        "path": "docker",
        "arguments": "start postgres-db"
      }
    }
  ]
}

Network Chaos with tc (Traffic Control)

# Add 200ms latency to all outgoing traffic (Linux)
sudo tc qdisc add dev eth0 root netem delay 200ms

# Add packet loss (10%)
sudo tc qdisc add dev eth0 root netem loss 10%

# Add both: 100ms delay with 20% jitter + 5% packet loss
sudo tc qdisc add dev eth0 root netem delay 100ms 20ms loss 5%

# Remove all chaos
sudo tc qdisc del dev eth0 root

# Verify current rules
tc qdisc show dev eth0

Chaos Testing with Playwright — Service Interruption

import subprocess
import time
import requests
import pytest

class ChaosController:
    """Controls chaos injection for testing"""

    def stop_service(self, service: str):
        subprocess.run(["docker", "stop", service], check=True)

    def start_service(self, service: str):
        subprocess.run(["docker", "start", service], check=True)

    def add_latency(self, ms: int, interface: str = "lo"):
        subprocess.run(
            ["tc", "qdisc", "add", "dev", interface, "root", "netem", "delay", f"{ms}ms"],
            check=True
        )

    def remove_latency(self, interface: str = "lo"):
        subprocess.run(["tc", "qdisc", "del", "dev", interface, "root"])


@pytest.fixture
def chaos():
    controller = ChaosController()
    yield controller
    # Teardown: ensure everything is restored
    try:
        controller.start_service("postgres-db")
        controller.remove_latency()
    except Exception:
        pass


def test_api_graceful_degradation_on_db_failure(chaos):
    """System should return 503 with proper error message when DB is down"""
    # Verify system is healthy first
    resp = requests.get("http://localhost:8080/health", timeout=5)
    assert resp.status_code == 200, "Precondition: system must be healthy"

    # Inject chaos: kill database
    chaos.stop_service("postgres-db")
    time.sleep(2)  # allow health checks to detect failure

    # System should degrade gracefully
    resp = requests.get("http://localhost:8080/api/users", timeout=5)
    assert resp.status_code == 503
    assert "service_unavailable" in resp.json().get("error", "").lower()
    assert "retry_after" in resp.headers, "Should tell clients when to retry"

    # Recover and verify
    chaos.start_service("postgres-db")
    time.sleep(5)  # allow reconnection

    resp = requests.get("http://localhost:8080/health", timeout=10)
    assert resp.status_code == 200, "System must fully recover"


def test_high_latency_shows_loading_state(chaos):
    """UI should show loading spinner when API is slow"""
    from playwright.sync_api import sync_playwright

    # Inject 3 second latency
    chaos.add_latency(3000)

    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto("http://localhost:3000/users")

        # Loading spinner should appear
        spinner = page.locator('[data-testid="loading-spinner"]')
        assert spinner.is_visible(), "Loading spinner must show during slow API"

        # Wait for content (extended timeout due to chaos)
        page.wait_for_selector('[data-testid="user-list"]', timeout=10000)
        browser.close()

Resilience Testing Checklist

CHAOS SCENARIOS TO TEST FOR EVERY SERVICE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Infrastructure Failures:
  [ ] Database down -> graceful 503 with retry-after header
  [ ] Cache (Redis) down -> fallback to DB, no 500s
  [ ] External API timeout -> circuit breaker triggers
  [ ] Message queue down -> requests queue locally

Network Chaos:
  [ ] 200ms added latency -> UI shows loading state
  [ ] 2s added latency -> timeouts handled, no hung requests
  [ ] 10% packet loss -> retries work, idempotent operations safe
  [ ] Total network partition -> cached data served, queue drains on recovery

Process Failures:
  [ ] Kill 1 of 3 app instances -> load balancer reroutes
  [ ] OOM kill on app -> pod restarts, in-flight requests gracefully failed
  [ ] Slow consumer -> backpressure handled

Data Corruption:
  [ ] Malformed JSON in queue -> dead letter queue receives, no crash
  [ ] Null values in required fields -> validation catches at boundary

Recovery:
  [ ] RTO (Recovery Time Objective): service healthy within 30s
  [ ] RPO (Recovery Point Objective): no data loss for committed transactions