42OpenTelemetry & Grafana

42.1 Three pillars of observability

42.2 OpenTelemetry

OTel is the open standard (CNCF) for generating telemetry. One SDK in your app emits traces+metrics+logs in OTLP format; you can swap backends without rewriting instrumentation.

Components

• SDK — library you import in your app to create spans/metrics
• Auto-instrumentation — JS/Python/Java agents that wrap common libs (Express, http, pg, kafka) without code changes
• Collector — separate process that receives OTLP, processes, exports to backends
• Exporters — backend-specific writers (OTLP, Prometheus, Jaeger, Datadog)

42.3 Spans and traces

// Node manual instrumentation
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('orders-service');

async function placeOrder(input) {
  const span = tracer.startSpan('placeOrder');
  span.setAttribute('user.id', input.userId);
  try {
    const order = await db.create(input);
    span.setAttribute('order.id', order.id);
    return order;
  } catch (e) {
    span.recordException(e);
    span.setStatus({ code: 2 /* ERROR */ });
    throw e;
  } finally {
    span.end();
  }
}

A trace = a tree of spans sharing the same traceId. Propagation happens via the traceparent HTTP header (W3C Trace Context standard). Downstream services join the same trace.

42.4 Collector pipeline

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 1s
  attributes:
    actions:
      - key: env
        value: production
        action: insert

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch, attributes], exporters: [otlp/tempo] }
    metrics: { receivers: [otlp], processors: [batch],            exporters: [prometheus]  }
    logs:    { receivers: [otlp], processors: [batch],            exporters: [loki]        }

42.5 Grafana

Visualisation layer. Connects to Prometheus (metrics), Loki (logs), Tempo (traces), and others. Dashboards combine all three in one view.

PromQL essentials

rate(http_requests_total[5m])                                # req/s
sum by (status) (rate(http_requests_total[5m]))              # by status
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # p95
1 - (sum(rate(http_requests_total{status=~"5.."}[5m])) /
     sum(rate(http_requests_total[5m])))                     # availability

42.6 SLI / SLO / error budget

If error budget is spent for the month, new feature deploys halt — only reliability work continues until next window. Mechanism to balance velocity vs reliability.

42.7 SDET use cases for OTel

• Assert that a test request produced a specific span graph (right services were called, in the right order, with expected attributes)
• Verify error spans on failure paths
• Check that latency stays within SLO budget under load
• Use trace ids in test output to link a failed test to the production trace UI

test('order flow generates correct trace', async ({ request }) => {
  const r = await request.post('/api/orders', {
    data: {/*...*/},
    headers: { 'x-test-correlation': 'abc-123' },
  });
  expect(r.status()).toBe(201);

  // Wait for trace ingestion
  await expect.poll(async () => {
    const traces = await fetch('http://tempo:3200/api/search?tags=test=abc-123').then(r => r.json());
    return traces?.traces?.length ?? 0;
  }, { timeout: 10_000 }).toBeGreaterThan(0);

  const trace = await fetch('http://tempo:3200/api/traces/...').then(r => r.json());
  const spans = trace.batches.flatMap(b => b.scopeSpans).flatMap(s => s.spans);
  const names = spans.map(s => s.name);
  expect(names).toContain('checkout-handler');
  expect(names).toContain('charge-stripe');
  expect(names).toContain('email-confirmation');
});

42.8 Logs vs metrics vs traces — when each?