66System Design Pro — Complete Reference

66.1 Interview framework

1. Clarify (5 min): users, scale, primary use cases, SLA, regions, mobile vs web, read/write ratio
2. Capacity estimate (5 min): DAU → QPS, storage, bandwidth. Use orders of magnitude.
3. API contracts (5 min): top endpoints, request/response shapes
4. Data model (5 min): entities + relationships + primary access pattern
5. High-level diagram (10 min): client → LB → app → DB → cache → queue → workers → CDN
6. Deep dive (15–20 min): pick hardest component; sharding, replication, caching, queue patterns
7. Scale + failures (10 min): 10× growth, region outage, hot key, cascading failure
8. Trade-offs throughout: SQL vs NoSQL, push vs pull, sync vs async, strong vs eventual

66.2 Capacity math (memorise)

Latency numbers everyone should know

66.3 Building blocks (each deep)

Load Balancer

• L4 (TCP): routes by IP/port. Fast, dumb. Used for non-HTTP (DBs, custom protocols).
• L7 (HTTP): routes by path/header/cookie. SSL termination. A/B routing. Body inspection.
• Algorithms: round-robin, least-connections, IP hash (sticky), weighted, EWMA.
• Health checks: HTTP /health endpoint; remove unhealthy from pool.
• Tools: Nginx, HAProxy, Envoy, AWS ALB/NLB, Cloudflare.
• Global LB: GSLB / anycast / DNS-based routing for geo distribution.

CDN

• Edge servers cache content close to users. Reduce latency + origin load.
• Push CDN: you upload assets to CDN proactively.
• Pull CDN: edge fetches on first miss, caches by Cache-Control.
• Edge compute: run code at edge (Cloudflare Workers, Lambda@Edge).
• Invalidation: tags or paths; can take seconds-minutes globally.
• Use cases: static assets, images, videos (HLS/DASH), API responses (with caution).

Cache

• Local in-process (LRU map): fastest; per-instance only.
• Distributed: Redis, Memcached. Shared across instances.
• Patterns:
  • Cache-aside (lazy): read miss → fetch DB → set cache. Most common.
  • Write-through: write hits cache + DB.
  • Write-back (write-behind): write to cache; flush to DB later. Risky.
  • Read-through: cache layer wraps DB; app talks to cache only.
  • Refresh-ahead: prefetch hot keys before expiry.
• Eviction: LRU, LFU, ARC, TTL-based.
• Hot key: single key hit by many → use multi-tier cache or replicate the key.
• Thundering herd: many clients miss same key simultaneously → single-flight pattern (one fetches, others wait).
• Cache stampede protection: probabilistic early expiration; XFetch algorithm.

Message Queue

• Point-to-point (SQS, RabbitMQ default): one message → one consumer.
• Pub/Sub (SNS, Redis Pub/Sub): one → many.
• Streaming (Kafka, Kinesis): durable log; consumers track offsets; replay possible.
• At-most-once: fire and forget; possible loss.
• At-least-once: redeliver on ack timeout; possible duplicates → idempotent consumer.
• Exactly-once: hard; needs transactional producer + idempotent consumer or distributed transaction.
• Backpressure: queue depth alarms; reject producers or shed load when consumers fall behind.
• DLQ (dead-letter queue): messages that fail processing N times → DLQ for human review.

Search Engine

• Inverted index: term → doc list. Built offline / streaming.
• BM25: relevance scoring (TF-IDF improved).
• Elasticsearch / OpenSearch / Meilisearch / Typesense.
• Sharded by doc ID hash for horizontal scale.
• Real-time: refresh interval (1s typical) vs throughput trade-off.
• Hybrid: text (BM25) + vector (semantic) — combine via reciprocal rank fusion.

Vector DB

• Stores embeddings; ANN search via HNSW / IVF / PQ.
• pgvector (Postgres ext), Pinecone, Weaviate, Qdrant, Milvus, Chroma.
• Used for: semantic search, RAG, recommendation, dedup.
• Metadata filtering + hybrid retrieval critical for production.

Object Storage

• S3 / GCS / Azure Blob: blob storage, REST API, 11 9s durability.
• Tiering: Standard → IA → Glacier (cost vs retrieval time).
• Multipart upload: parallel chunks for large files.
• Presigned URLs: client uploads/downloads directly (skip app server).
• Versioning: keep history; protect against accidental delete.
• Event triggers: S3 → Lambda → process new files.

66.4 Database internals (deep)

B-tree (Postgres, MySQL default)

• Self-balancing tree; each node holds N keys (sorted).
• Reads: O(log_N(rows)) — typically 3-4 disk accesses for billions of rows.
• Good for reads + range scans.
• Writes: page splits when full → write amplification.
• WAL (write-ahead log) ensures durability + crash recovery.

LSM tree (Cassandra, RocksDB, LevelDB)

• Writes go to in-memory memtable; flushed as immutable SSTables on disk.
• Reads check memtable + bloom-filter-gated SSTables.
• Compaction: merge SSTables (read amplification, write amplification, space amplification trade-off).
• Optimised for writes; reads slower than B-tree (multiple SSTable lookups).

MVCC (Multi-Version Concurrency Control)

• Each row has multiple versions tagged with transaction id.
• Readers don't block writers; writers don't block readers.
• Postgres: in-place updates create new tuple; old tuple garbage-collected by VACUUM.
• Cost: bloat if VACUUM falls behind; tuning autovacuum is critical at scale.

Isolation levels (deep)

Replication protocols

• Async: leader acks client immediately; replicates later. Fast; risk of data loss on leader crash.
• Semi-sync: leader waits for at least one replica ack. Compromise.
• Sync: leader waits for all replicas. Strong durability; slow.
• Logical replication: replicates row-level changes (CDC use case).
• Physical replication: replicates WAL bytes; faster, fewer features.

Sharding

• Range: by key range. Easy range scans; hot spots if data skewed.
• Hash: by hash(key) % N. Uniform; bad for range queries.
• Consistent hash: ring of nodes; key goes to nearest clockwise. Rebalancing moves ~1/N of keys.
• Directory-based: explicit lookup table key → shard. Flexible; extra hop + SPOF.
• Geographic: by region (regulatory / locality).
• Cross-shard queries are expensive — design schema to fit access patterns.

66.5 Distributed systems (deep)

CAP theorem (revisited)

In a partition, choose Consistency or Availability. Partition is unavoidable in real systems → effectively choose CP or AP.

• CP: Postgres, MongoDB (with majority writes), ZooKeeper. Refuses writes during partition.
• AP: Cassandra, DynamoDB, CouchDB. Accepts writes; eventual consistency via repair.

PACELC

Extends CAP. If Partition: A vs C. Else: Latency vs Consistency. Captures the always-on trade-off, not just partition behaviour.

Consistency models (full hierarchy, strongest → weakest)

1. Linearizable (atomic): every op appears instantaneously between invocation and response.
2. Sequential: all clients see same global order; not necessarily real-time.
3. Causal: causally related operations seen in same order; concurrent ops can differ.
4. Read-your-writes: user sees own writes.
5. Monotonic reads: never see older value once you've seen newer.
6. Monotonic writes: writes from same client applied in order.
7. Eventual: replicas converge given no writes.

Consensus algorithms

Paxos (Lamport, 1989)

• Roles: Proposer, Acceptor, Learner.
• Two phases: Prepare (get promise from majority) + Accept (propose value).
• Hard to understand; basis for many real systems.

Raft (Ongaro & Ousterhout, 2014)

• Designed for understandability.
• Leader election (heartbeats; if absent → candidate requests votes; majority wins).
• Log replication: leader appends; majority ack → committed.
• Safety: log matching property guarantees consistency.
• Used by: etcd, Consul, CockroachDB, TiDB.

ZAB (ZooKeeper Atomic Broadcast)

• Similar to Raft; predates it.
• Used by ZooKeeper.

Vector clocks

Per-node counter, exchanged with each message. Compare two vectors:

• V1 ≤ V2 if every entry V1[i] ≤ V2[i] → V1 happened-before V2.
• Neither ≤ → concurrent.

Used by Dynamo, Voldemort to detect conflicts; client / app must resolve.

CRDTs (Conflict-Free Replicated Data Types)

• Data structures where concurrent updates automatically merge without conflict.
• G-Counter: grow-only counter; each replica increments its own slot; total = sum.
• PN-Counter: positive + negative counters.
• G-Set: grow-only set; union on merge.
• OR-Set: observed-remove set; supports adds + removes correctly.
• LWW-Register: last-writer-wins (timestamp).
• Used by Riak, Redis CRDT, collaboration tools (Yjs, Automerge).

Gossip protocols

• Each node periodically picks K random peers and exchanges state.
• Information spreads exponentially; converges in O(log N).
• Used for: cluster membership (Cassandra, Consul), failure detection.

Two-phase commit (2PC)

• Coordinator: phase 1 (prepare) — ask participants to vote. Phase 2 (commit/abort) — based on votes.
• Blocking: coordinator crash during phase 2 stalls participants.
• Used rarely now; sagas preferred for microservices.

Saga pattern

• Distributed transaction as chain of local transactions + compensating actions.
• If step N fails, run inverse of steps 1..N-1.
• Orchestration: central state machine (Temporal, Camunda, AWS Step Functions).
• Choreography: services react to events; no central coord.

66.6 Networking deep

TCP

• 3-way handshake: SYN → SYN-ACK → ACK.
• Reliable, ordered, byte stream.
• Congestion control: slow start → AIMD (additive increase, multiplicative decrease) → modern: CUBIC, BBR.
• Head-of-line blocking: lost packet stalls entire stream.

HTTP/1.1 vs HTTP/2 vs HTTP/3

gRPC

• HTTP/2 + Protobuf. Strong typing, efficient binary, bidi streaming.
• Code generation from .proto in many languages.
• Server reflection for dynamic discovery.
• gRPC-Web for browser support (proxy needed).

WebSocket

• HTTP upgrade → persistent bidirectional TCP.
• Used for chat, live updates, collaboration.
• Scaling: sticky sessions + Redis pub/sub or Kafka for fanout.
• Heartbeats to detect dead connections.

SSE (Server-Sent Events)

• One-way server → client stream over HTTP.
• Auto-reconnect built in.
• Simpler than WebSocket when bidirectional not needed (LLM token streaming, notifications).

TLS handshake (TLS 1.3)

• 1-RTT: client hello (key share) → server hello (key share + cert) → both compute shared secret via ECDHE → finished.
• 0-RTT for resumed sessions (replay-attack risk; restrict to idempotent ops).
• PFS (perfect forward secrecy) — keys not derivable from long-term cert key.

DNS

• Resolver → root → TLD → authoritative.
• Records: A, AAAA, CNAME, MX, TXT, SRV.
• TTL controls cache duration. Low TTL = fast change, more queries.
• Anycast for global low-latency resolvers.
• DNS-based failover for GSLB.

66.7 Failure handling toolkit

• Timeout: every external call; pick based on p99 + buffer; cascade them downward.
• Retry: exponential backoff + jitter; only idempotent ops; max attempts.
• Circuit breaker: closed → open (fail fast) → half-open (probe).
• Bulkhead: separate thread pools / connection pools per downstream so one failure doesn't drain everything.
• Rate limit / load shed: refuse traffic when overloaded; preserve core functions.
• Graceful degradation: serve cached / partial data on dep failure.
• Fallback: alternative path or provider.
• Dead-letter queue: park poison messages for triage.
• Health check: liveness (am I alive?) + readiness (am I ready to serve?).
• Chaos testing: prove all of the above actually work.

66.8 Worked end-to-end designs (15+)

(1) URL shortener (e.g. bit.ly)

Scale: 100M URLs created/year → 3 URLs/sec. Reads dominate writes ~100:1.

• POST /shorten {url} → 7-char base62 slug; 62^7 ≈ 3.5T URLs.
• Slug generation: counter (centralised, predictable) OR random hash (avoid collisions via retry).
• Storage: KV (slug → {url, expiry, owner, created_at}). Postgres or DynamoDB.
• Cache hot slugs in Redis (LRU, TTL).
• GET /:slug → 301 redirect (cacheable in browser).
• Click analytics: async fire-and-forget to Kafka → warehouse.
• Rate limit per IP; abuse / malware detection.
• Custom slugs (vanity URL) — uniqueness check.

(2) Distributed unique id generator (Snowflake)

64-bit ID:

• 1 sign bit (unused)
• 41 timestamp bits (ms since epoch) — ~69 years
• 10 node bits — 1024 nodes
• 12 sequence bits — 4096 ids per ms per node

Time-sortable, no central coord. Modern: UUIDv7 (similar guarantees, standard).

(3) Rate limiter

• Token bucket: refill rate R; bucket size N. Allows bursts up to N.
• Leaky bucket: smooths output at fixed rate.
• Sliding window log: store timestamps in sorted set. Precise; memory-heavy.
• Sliding window counter: weighted average of current + previous fixed windows. Approximate; cheap.
• Implementation: Redis INCR + EXPIRE; Lua script for atomicity.
• Distributed: per-edge enforcement + central Redis source of truth.
• Response: 429 + Retry-After header.

(4) Chat (1-1 + group)

Scale: 1B users; 50 msgs/user/day = 50B/day = 580K msgs/sec average.

• WS edge cluster + sticky sessions.
• Connection-manager service tracks userId → connectionId (Redis).
• Message: producer publishes to Kafka per channel.
• Recipient's connection-manager subscribes; pushes via WS.
• Persistence: Cassandra (write-heavy, time-partitioned by channel).
• Read receipts: separate events.
• Presence: Redis SET online_users with TTL refreshed by heartbeat.
• Group chat: write once to channel topic; all subscribers see.
• End-to-end encryption (Signal): client-side; server stores ciphertext.

(5) News feed (Twitter / X)

• Fanout-on-write: when user posts, write to each follower's feed cache.
  • Pro: cheap reads (just fetch follower's feed).
  • Con: expensive for celebs with millions of followers; lots of duplicate storage.
• Fanout-on-read: when user opens feed, query latest posts from accounts they follow + merge.
  • Pro: no precomputation.
  • Con: expensive read query, especially for users following many.
• Hybrid: fanout-on-write for normal users; fanout-on-read for celebs (defined by follower threshold).
• Feed cache: Redis sorted set per user (score = post timestamp).
• Ranking: ML model re-ranks on read (chronological is now rare).
• Media: S3 + CDN; multiple resolutions transcoded async.

(6) Uber-like ride matching

• Geo-index drivers: S2 cells / Quad tree / Geohash buckets in Redis.
• Driver app sends location updates every 4 seconds via WS / UDP.
• Rider request: query nearby drivers in K cells around pickup; pick by ETA + rating.
• Trip state machine: requested → matched → en-route → started → completed.
• Payment as async post-trip (Stripe).
• Surge pricing: based on supply/demand per cell; recomputed every few minutes.
• Reliability: each request retried; failover regions.

(7) YouTube / video streaming

• Upload → S3.
• Transcoding pipeline (queue → workers): produce multiple bitrates (240p, 480p, 720p, 1080p, 4K).
• HLS / DASH adaptive streaming; CDN serves segments.
• Metadata in Postgres (title, owner, upload time).
• Views / likes in Cassandra (massive write throughput).
• Recommendation: offline batch ML + real-time signals (Kafka).
• Search: Elasticsearch.
• Live streaming: ingest → transcode → CDN with lower latency (LL-HLS).

(8) Dropbox-like file sync

• File chunked into 4 MB blocks; content-hash addressed (dedup across users).
• Block storage: S3.
• Metadata DB (Postgres): user → file → block list + versions.
• Sync: delta updates; client polls or WS for change notifications.
• Conflict resolution: last-writer-wins with version vectors; "conflicted copy" file created.
• End-to-end encryption (optional): client-side; server can't read content.

(9) Search autocomplete

• Trie of past queries; precompute top-K results per prefix node.
• Or Elasticsearch completion suggester.
• Cache hot prefixes in Redis.
• Real-time: streaming query log → update top-K per prefix periodically (every minute).
• Ranking signals: popularity, recency, personalisation, location.
• Typo tolerance: edit-distance ≤ K via fuzzy search.

(10) Distributed cache (Memcached / Redis cluster)

• Consistent hash for shard selection.
• Replication factor 2-3 for fault tolerance.
• Eviction: LRU default; LFU for frequency-aware.
• TTL on writes; lazy expiration on read.
• Cache warming after restart (replay from log or migrate from peer).
• Hot-key protection: read-through with single-flight; replicate hot keys.

(11) Distributed file system (HDFS / GFS)

• Namenode (master): metadata (file → blocks → datanodes); SPOF (HA via ZooKeeper).
• Datanode (worker): stores blocks (128 MB default); 3× replication.
• Write: client → namenode (allocates block) → write to 3 datanodes (pipeline).
• Read: client → namenode (get block locations) → read from nearest datanode.
• Rack-aware placement: 1 replica local, 1 same rack, 1 different rack.

(12) Notification system

• Multi-channel: push, email, SMS, in-app.
• Producer publishes notification intent → queue.
• Worker fans out to channel-specific workers based on user prefs + opt-outs.
• Per-channel provider (FCM/APNs for push; SES/SendGrid for email; Twilio for SMS).
• Rate limit per user; quiet hours; A/B test templates.
• Idempotency key dedupes retries.
• DLQ for failed sends.

(13) Payment system

• API gateway → Payment service → Stripe / Adyen.
• Idempotency key required (UUID per checkout attempt).
• Webhook from provider for async outcomes (3DS, refunds).
• Ledger: append-only Postgres / event sourcing for audit.
• Reconciliation jobs: nightly compare our records vs provider's.
• Fraud detection: rules + ML on transaction stream.
• PCI-DSS scope minimization: tokenize cards via provider; never store raw PAN.

(14) Ads serving

• Auction: real-time bidding (RTB) — < 100 ms response per request.
• Index ads with targeting criteria (audience, geo, time).
• Candidate generation: select eligible ads (cheap).
• Ranking: ML model predicts CTR × bid → score.
• Frequency capping: per user/day (Redis counters).
• Budget pacing: distribute budget through the day.
• Click-through tracking; attribution (last-click, multi-touch).

(15) Recommendation engine (Netflix / YouTube)

• Collaborative filtering (user-item matrix, matrix factorization).
• Content-based (item features).
• Hybrid + deep learning rankers.
• Offline batch training (Spark, GPU clusters); features in feature store.
• Real-time scoring at request time.
• Cold start: fall back to popular / trending / similar users.

(16) Real-time analytics / dashboards

• Events → Kafka → stream processor (Flink / Spark Streaming).
• Aggregations (count, sum, percentiles) → time-series DB (ClickHouse, Druid, InfluxDB).
• Approximate algorithms (HyperLogLog, T-Digest) for cardinality / quantiles at scale.
• Dashboard queries served from precomputed cubes.

66.9 Architectural styles

66.10 Anti-patterns + scaling pitfalls

• Distributed monolith: microservices that must deploy together. Worst of both worlds.
• Shared database across services: tight coupling, can't evolve schema independently. Anti-microservice.
• Synchronous chain of N services: latency + failure compounds. Use events.
• Dual-write: write to DB + Kafka non-transactionally. Use outbox pattern.
• Cache-without-TTL: stale data forever. Always TTL or explicit invalidation.
• SELECT * in production: bandwidth + breaks on schema change.
• N+1 queries: classic ORM gotcha. Eager-load or DataLoader.
• No backpressure: producers overwhelm consumers; queue grows unbounded.
• Optimistic without conflict resolution: silent data loss.
• Pet servers (vs cattle): hand-configured instances; can't be replaced. Use IaC.
• Ignoring p99 / tail latency: averages lie. Optimise for the slowest 1%.
• Premature microservices: split a monolith you don't yet understand → split wrong.

66.11 Real-world case-study takeaways

Netflix

• Open Connect: own CDN appliances inside ISPs for video delivery.
• Hystrix → Resilience4j: circuit breakers everywhere.
• Chaos Monkey: continuous failure injection in prod.
• Cassandra for time-series (viewing history); evcache (memcached fork) for caching.

Twitter / X

• Hybrid timeline (precomputed for normal, on-read for celebs).
• Snowflake: distributed id generator.
• Manhattan: in-house multi-tenant K/V store.

WhatsApp

• Erlang for massive concurrency (per-user process model).
• 50 engineers serving 900M users at peak (pre-Meta).
• BEAM VM's per-process GC = no global pauses.

Discord

• Elixir/Phoenix for realtime.
• Migrated MongoDB → Cassandra → ScyllaDB for messages (write throughput).
• Voice: WebRTC + own SFU servers.

Stripe

• Idempotency-key pattern for safe retries.
• API versioning by date; old versions kept alive years.
• Heavy investment in observability + automated remediation.

Amazon DynamoDB

• Single-table design: model access patterns into a single denormalised table with composite keys.
• GSIs (Global Secondary Indexes) for alternate access patterns.
• Eventually consistent reads default; strongly consistent reads cost double.

66.X Worked Designs — Code Reference

URL Shortener — Base62 + FastAPI

from fastapi import FastAPI, HTTPException
from fastapi.responses import RedirectResponse
import redis, string, random

app = FastAPI()
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
CHARS = string.ascii_letters + string.digits  # 62 chars

def base62_encode(n: int) -> str:
    result = ""
    while n:
        result = CHARS[n % 62] + result
        n //= 62
    return result or CHARS[0]

def generate_short_code(url: str) -> str:
    # Use Redis INCR for monotonic ID
    seq_id = r.incr("url:seq")
    code = base62_encode(seq_id)
    r.setex(f"url:{code}", 86400 * 30, url)
    return code

@app.post("/shorten")
def shorten(url: str) -> dict:
    # Check existing (dedup by URL hash)
    import hashlib
    url_hash = hashlib.sha256(url.encode()).hexdigest()[:16]
    existing = r.get(f"hash:{url_hash}")
    if existing:
        return {"short_code": existing, "url": url}
    code = generate_short_code(url)
    r.set(f"hash:{url_hash}", code)
    return {"short_code": code, "url": url}

@app.get("/{code}")
def redirect(code: str):
    url = r.get(f"url:{code}")
    if not url:
        raise HTTPException(status_code=404, detail="Link not found or expired")
    r.incr(f"url:{code}:clicks")  # analytics
    return RedirectResponse(url=url, status_code=302)

Chat Service — WebSocket + Redis Pub/Sub

from fastapi import FastAPI, WebSocket, WebSocketDisconnect
import redis.asyncio as aioredis
import asyncio, json

app = FastAPI()

class ConnectionManager:
    def __init__(self):
        self.rooms: dict[str, list[WebSocket]] = {}

    async def connect(self, ws: WebSocket, room: str):
        await ws.accept()
        self.rooms.setdefault(room, []).append(ws)

    def disconnect(self, ws: WebSocket, room: str):
        self.rooms.get(room, []).remove(ws)

    async def broadcast(self, room: str, message: str, sender: WebSocket = None):
        for ws in self.rooms.get(room, []):
            if ws is not sender:
                await ws.send_text(message)

mgr = ConnectionManager()
redis_pub = aioredis.Redis(host="localhost")
redis_sub = aioredis.Redis(host="localhost")

@app.websocket("/ws/{room}/{user_id}")
async def websocket_endpoint(ws: WebSocket, room: str, user_id: str):
    await mgr.connect(ws, room)
    sub = redis_sub.pubsub()
    await sub.subscribe(f"chat:{room}")

    async def listen_redis():
        async for msg in sub.listen():
            if msg["type"] == "message":
                await ws.send_text(msg["data"])

    asyncio.create_task(listen_redis())
    try:
        while True:
            data = await ws.receive_text()
            payload = json.dumps({"user": user_id, "text": data})
            await redis_pub.publish(f"chat:{room}", payload)
    except WebSocketDisconnect:
        mgr.disconnect(ws, room)

News Feed Fanout — Write Path

import redis, json, time

r = redis.Redis(host="localhost", port=6379)

def post_tweet(user_id: int, content: str) -> int:
    tweet_id = r.incr("tweet:seq")
    tweet = {"id": tweet_id, "user_id": user_id, "content": content, "ts": int(time.time())}
    r.set(f"tweet:{tweet_id}", json.dumps(tweet))

    # Fan-out to followers (write-to-fan-out for small follower count)
    followers = r.smembers(f"followers:{user_id}")
    pipe = r.pipeline()
    for follower_id in followers:
        # Each follower has a sorted set timeline, score = timestamp
        pipe.zadd(f"timeline:{follower_id.decode()}", {str(tweet_id): tweet["ts"]})
        pipe.zremrangebyrank(f"timeline:{follower_id.decode()}", 0, -1001)  # keep last 1000
    pipe.execute()
    return tweet_id

def get_feed(user_id: int, page: int = 0, per_page: int = 20) -> list:
    start = page * per_page
    end   = start + per_page - 1
    tweet_ids = r.zrevrange(f"timeline:{user_id}", start, end)
    pipe = r.pipeline()
    for tid in tweet_ids:
        pipe.get(f"tweet:{tid.decode()}")
    return [json.loads(t) for t in pipe.execute() if t]

Distributed Lock — Redis SETNX

import redis, uuid, time
from contextlib import contextmanager

r = redis.Redis(host="localhost", port=6379)

@contextmanager
def distributed_lock(resource: str, ttl: int = 30, retry: int = 3, wait: float = 0.1):
    lock_key = f"lock:{resource}"
    lock_val = str(uuid.uuid4())
    acquired = False
    for _ in range(retry):
        acquired = r.set(lock_key, lock_val, nx=True, ex=ttl)
        if acquired:
            break
        time.sleep(wait)
    if not acquired:
        raise Exception(f"Could not acquire lock on {resource}")
    try:
        yield
    finally:
        # Release only if still our lock (Lua for atomicity)
        script = """
        if redis.call("get",KEYS[1]) == ARGV[1] then
            return redis.call("del",KEYS[1])
        else return 0 end
        """
        r.eval(script, 1, lock_key, lock_val)

# Usage
# with distributed_lock("payment:order_99"):
#     process_payment("order_99")

Bloom Filter — Duplicate URL Detection

import math, mmh3
from bitarray import bitarray

class BloomFilter:
    def __init__(self, n: int, fp_rate: float = 0.01):
        m = int(-n * math.log(fp_rate) / (math.log(2) ** 2))
        k = int((m / n) * math.log(2))
        self.m = m
        self.k = k
        self.bits = bitarray(m)
        self.bits.setall(0)

    def add(self, item: str):
        for seed in range(self.k):
            idx = mmh3.hash(item, seed) % self.m
            self.bits[idx] = 1

    def __contains__(self, item: str) -> bool:
        return all(self.bits[mmh3.hash(item, s) % self.m] for s in range(self.k))

# URL deduplication for web crawler
seen = BloomFilter(n=10_000_000, fp_rate=0.001)  # 10M URLs, 0.1% FP rate
seen.add("https://example.com/page1")
print("https://example.com/page1" in seen)  # True
print("https://example.com/page2" in seen)  # False