51Microservices & Event-Driven (Kafka) Testing

51.1 Microservices testing pyramid

1. Unit — within one service
2. Component — service in isolation with real DB, mocked deps
3. Contract — Pact / Spring Cloud Contract — cross-service expectations (Module 49)
4. Integration — small graph of services together
5. End-to-end — full system, minimal
6. Synthetic monitoring — prod heartbeat (Module 30)

51.2 Kafka in 5 minutes

• Topic — named append-only log
• Partition — topic split into ordered shards for parallelism
• Producer — writes records to a partition
• Consumer — reads records sequentially; tracks its offset
• Consumer group — multiple consumers split the partitions (parallel processing of one topic)
• Retention — records expire after N hours/MB (default 7 days)
• Broker — Kafka server node; cluster = many brokers

51.3 Producing + consuming from tests (Node)

import { Kafka } from 'kafkajs';
const kafka = new Kafka({ clientId: 'test', brokers: ['localhost:9092'] });

test('publishing an order triggers an email', async () => {
  // Produce
  const producer = kafka.producer();
  await producer.connect();
  await producer.send({
    topic: 'orders.created',
    messages: [{ key: 'ord-123', value: JSON.stringify({ id: 'ord-123', total: 99 }) }],
  });
  await producer.disconnect();

  // Verify side effect via DB / downstream API / consume another topic
  const consumer = kafka.consumer({ groupId: 'test-' + Date.now() });
  await consumer.connect();
  await consumer.subscribe({ topic: 'emails.sent', fromBeginning: false });
  let received: any = null;
  await consumer.run({ eachMessage: async ({ message }) => {
    const v = JSON.parse(message.value!.toString());
    if (v.orderId === 'ord-123') received = v;
  }});
  await expect.poll(() => received, { timeout: 15_000 }).not.toBeNull();
  expect(received.subject).toContain('Your order ord-123');
  await consumer.disconnect();
});

51.4 Schema registry + Avro/Protobuf

Producers and consumers agree on a binary schema; the registry stores versions. Why it matters for tests:

• Backward/forward compatibility rules prevent breaking changes
• Tests can verify a producer's payload still satisfies the registered schema
• Consumers can be tested against old + new schema versions

51.5 Async assertion patterns

// expect.poll — retry an async check
await expect.poll(async () => {
  const r = await db.query("SELECT status FROM orders WHERE id = $1", ['ord-1']);
  return r.rows[0]?.status;
}, { timeout: 10_000, intervals: [200, 500, 1000] }).toBe('paid');

// Awaitable event helper
async function awaitMessage(topic: string, match: (v: any) => boolean, timeoutMs = 10_000) {
  return new Promise((resolve, reject) => {
    const consumer = kafka.consumer({ groupId: `t-${Date.now()}` });
    const timer = setTimeout(() => reject(new Error('timeout')), timeoutMs);
    (async () => {
      await consumer.connect();
      await consumer.subscribe({ topic, fromBeginning: false });
      await consumer.run({ eachMessage: async ({ message }) => {
        const v = JSON.parse(message.value!.toString());
        if (match(v)) { clearTimeout(timer); await consumer.disconnect(); resolve(v); }
      }});
    })();
  });
}

51.6 Idempotency & retries

• At-least-once delivery is the default — your consumer might process the same message twice. Always design idempotent handlers.
• Test it: send the same message twice with the same key, verify the side effect happens once.
• Dead-letter queues (DLQ) — messages that fail processing N times go to a separate topic for human review.
• Test DLQ: produce a poison message, verify it lands in the DLQ after retry budget exhausts.

51.7 Eventual consistency

In an event-driven system the DB write happens, then events propagate to downstream services and update their projections. Tests must wait for that propagation — never sleep for a fixed time; poll on the observable end state.

51.8 Tools

51.9 Service mesh / observability

• Distributed tracing (Jaeger, Datadog APM, OpenTelemetry) — correlates a request across services via a trace id. Tests can assert that a request produced the expected span graph.
• Service mesh (Istio, Linkerd) — sidecar proxies handle retries, timeouts, mTLS. Tests verify policies (e.g. circuit breaker opens after N failures).

51.10 Chaos engineering for microservices

• Kill a pod (Chaos Mesh / Litmus) — verify retries handle it
• Inject latency to a downstream — verify timeouts + circuit breaker
• Drop a percentage of Kafka messages — verify reconciliation
• Run game days — full-team chaos exercises