Lesson 1: Caching

A reference-style deep dive into hit/miss ratios, eviction policies (LRU/LFU), invalidation, write-through vs write-behind. Read this as an article, not a transcript. The accompanying audio is the spoken companion; the article below is the canonical written reference.

Audience. Engineers designing, building, or operating distributed systems who want a clear mental model rather than a checklist of tools.

Prerequisites. Working knowledge of HTTP, basic SQL, and the idea of running more than one server behind a load balancer.

Why caching matters
The core mental model and where to start 3-12. In-depth sections below (see "Lesson body")

Lesson diagram

Lesson 1 diagram — Caching

Figure 1. The canonical caching topology and control flow covered in this lesson.

1. Why caching matters

Caching is the single biggest lever you have for latency and cost. Done right it absorbs 90% of read traffic at near-zero compute; done wrong it returns stale data and silently corrupts correctness. A cache is a contract: the system gets faster, and in exchange the user agrees that reads can be slightly stale. The trick is choosing how stale, and where to enforce the contract.

Every large system you use — Reddit, Twitter, Netflix, GitHub — leans heavily on caching. The shape of the cache is what determines the tail latency, the cost of capacity, and the worst-case behavior during traffic spikes. Caching is not a "later" optimization; it is a first-class design decision that shapes the rest of the architecture.

Three properties make caching special:

It is on the critical path. Every read request touches it directly or transitively. A slow cache is a slow application.
It shapes the failure modes. When the cache fails, the backend is exposed to 100% of traffic — usually more than it can absorb.
It is hard to retrofit. Choosing the wrong cache topology early forces expensive migrations later.

This lesson walks through the design space, the algorithms, and the operational consequences so you can choose correctly the first time.

2. Locality — the reason caches work

Caches work because real workloads exhibit locality. There are two flavors:

Temporal locality — recently-used data is likely to be used again soon. A loop variable is the canonical example: it is read and written on every iteration.
Spatial locality — data near recently-used data is likely to be used soon. An array scanned sequentially is the canonical example: when you read element i, you will probably read element i+1 next.

Both flavors are why your CPU has L1/L2/L3 caches, why your OS has a page cache, and why your database has a buffer pool. Hardware locality and software locality follow the same laws. The trick at the application layer is to recognize that every "hot" piece of data is hot because of locality, and design your cache around the locality pattern you actually have — not the one you wish you had.

# CPU-style locality example (C)
int sum = 0;
for (int i = 0; i < N; i++) {
    sum += array[i];   // spatial locality: array[i] soon reads array[i+1]
}

# App-style locality example (Python)
recent_users = collections.OrderedDict()    # LRU-ish
def get_user(uid):
    if uid in recent_users:
        recent_users.move_to_end(uid)        # touch
        return recent_users[uid]
    user = db.get_user(uid)
    recent_users[uid] = user
    if len(recent_users) > 10_000:
        recent_users.popitem(last=False)     # evict LRU
    return user

The CPU example exploits spatial locality in hardware. The app example exploits temporal locality in the database. The cache structure (a dict, in this case) is identical — what changes is the eviction policy and the source of truth on miss.

3. Anatomy of a cache

A cache has four moving parts, no matter where it sits:

The key space — what you look up by. Usually a hash of the request identity (user id, URL, query).
The store — where the cached values live. In-process (dict, Caffeine, Guava), in a separate process (Redis, Memcached), or on the edge (CDN).
The eviction policy — what to drop when full. LRU, LFU, ARC, FIFO, TTL-only.
The invalidation mechanism — how cached values get refreshed or removed. TTL, event-driven, write-through.

Each of these has its own trade-offs and failure modes. Choose them independently. A Redis cache with TTL-only invalidation is a different system from a Redis cache with event-driven invalidation; the data they serve has different staleness characteristics.

4. Eviction policies — the heart of the cache

The eviction policy determines what stays in the cache when memory is full. The four that matter:

LRU — Least Recently Used

Evict the entry that has not been accessed for the longest time. Best for: workloads with temporal locality. Worst for: scans over the keyspace that touch each entry once.

from collections import OrderedDict
class LRUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.cache = OrderedDict()
    def get(self, key):
        if key not in self.cache:
            return None
        self.cache.move_to_end(key)
        return self.cache[key]
    def put(self, key, value):
        if key in self.cache:
            self.cache.move_to_end(key)
        self.cache[key] = value
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False)

LRU is the default in most systems because it works well across a wide range of workloads. The cost is O(1) per access but requires bookkeeping (the linked list of recency).

LFU — Least Frequently Used

Evict the entry with the lowest access count. Best for: workloads with skewed access patterns (a few keys get 99% of the traffic). Worst for: shifting workloads (the new hot key has count 1 and gets evicted before it can prove itself).

import heapq
from collections import Counter
class LFUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.counts = Counter()
        self.cache = {}
    def get(self, key):
        if key not in self.cache:
            return None
        self.counts[key] += 1
        return self.cache[key]
    def put(self, key, value):
        if key in self.cache:
            self.counts[key] += 1
        else:
            if len(self.cache) >= self.capacity:
                # Evict the least frequent
                evict_key = min(self.cache, key=lambda k: self.counts[k])
                del self.cache[evict_key]
                del self.counts[evict_key]
            self.cache[key] = value
            self.counts[key] = 1

LFU is more memory-intensive than LRU but outperforms it dramatically when the workload has long-lived hot keys (think product catalog, user profiles).

ARC — Adaptive Replacement Cache

A hybrid that tracks both recency and frequency, and adapts the split between them based on observed workload. Used in IBM's research systems; the algorithm is patented but the idea influenced many successors.

TTL-only — Time-To-Live

No eviction policy at all; every entry expires after a fixed TTL. Best for: caches of external data where staleness is bounded by the upstream's update rate. Worst for: workloads with bursty access (a popular post may expire right when it's getting the most traffic).

5. Invalidation — the hardest problem

There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors. The joke is old but the lesson is fresh: invalidation is where caches go wrong.

Three strategies:

TTL — bound the staleness

cache.set(key, value, ttl=300)    # expire after 5 minutes

Simple, predictable, eventually consistent. The risk is that a TTL of 300s means reads can be up to 300s stale. Pick the smallest TTL your correctness allows.

Event-driven — invalidate on change

# When the underlying data changes:
db.update(...)
cache.invalidate(key)              # next read will miss and re-populate

Tighter consistency, more moving parts. You need a reliable event channel (Postgres logical replication, Kafka, Redis pub/sub). The risk is missed events — if the channel drops a message, the cache serves stale data forever.

Write-through — keep cache and store in sync

def update(key, value):
    db.set(key, value)
    cache.set(key, value)          # update cache atomically

Strongest consistency, highest write cost. Used when the cost of stale reads is unacceptable (e.g. account balances).

The right strategy depends on your consistency budget. If your users can tolerate 30 seconds of staleness, TTL is fine. If they cannot, you need event-driven or write-through, and you need to budget for the operational complexity.

6. Cache patterns in the application layer

The five patterns every backend engineer should know:

Cache-aside (lazy loading)

Application reads cache, falls back to store, populates cache.

def get_user(uid):
    user = cache.get(f"user:{uid}")
    if user is None:
        user = db.get_user(uid)
        cache.set(f"user:{uid}", user, ttl=300)
    return user

The most common pattern. Application owns the cache logic. Easy to reason about; works with any cache backend.

Read-through

The cache library itself fetches from the store on miss.

@cache.read_through(key_fn=lambda uid: f"user:{uid}", ttl=300)
def get_user(uid):
    return db.get_user(uid)

The application does not know about the cache. Cleaner application code; tightly coupled to a specific cache library.

Write-through

Cache and store are updated atomically on every write.

def update_user(uid, fields):
    db.update_user(uid, fields)
    cache.set(f"user:{uid}", db.get_user(uid))

Strong consistency. Higher write cost.

Write-behind (write-back)

Write to cache immediately; write to store asynchronously.

def update_user(uid, fields):
    cache.set(f"user:{uid}", {**cache.get(f"user:{uid}"), **fields})
    queue.enqueue("db_update", (uid, fields))   # flushed every 10s

Fast writes. Risk: if the cache dies before the queue flushes, the write is lost.

Write-around

Write to store; bypass cache. The next read populates the cache.

def update_user(uid, fields):
    db.update_user(uid, fields)
    # Do not touch cache — let next read populate

Good for write-heavy workloads where the written value is unlikely to be re-read soon. Avoids cache pollution from cold writes.

7. Distributed caches — Redis, Memcached, and beyond

Single-process caches work for one machine. Distributed caches work for many.

Redis

A single-threaded in-memory store with optional persistence. Supports rich data types (strings, hashes, lists, sets, sorted sets), pub/sub, scripting (Lua), transactions. The default choice for application-layer caching in 2026.

# Typical Redis usage
redis.set("user:42", json.dumps(user), ex=300)
user = json.loads(redis.get("user:42"))
redis.hincrby("post:99:likes", "", 1)         # atomic counter
redis.zadd("leaderboard", {user_id: score})   # sorted set

Redis Cluster shards data across nodes; each shard is a master + replicas. Built-in failover.

Memcached

A simpler in-memory key-value store. No persistence, no rich types, multi-threaded. Faster than Redis for pure get/set workloads but less versatile.

CDN edge caches

Cloudflare, CloudFront, Fastly. Cache HTTP responses at the geographic edge. Best for: static assets, API responses with cache-control headers, anything with high read-to-write ratio.

8. The thundering herd problem

When a hot key expires, every concurrent request misses at once. They all hit the backend simultaneously. The backend gets a spike it cannot absorb. This is the thundering herd (also: cache stampede).

Solutions:

Locking

def get(key):
    if cache.get(key):
        return cache.get(key)
    # Acquire lock so only one thread populates
    if lock.acquire(key, timeout=5):
        try:
            value = db.get(key)
            cache.set(key, value, ttl=300)
            return value
        finally:
            lock.release(key)
    else:
        # Another thread is populating; wait or fall back
        time.sleep(0.1)
        return cache.get(key) or db.get(key)

Early expiration (probabilistic)

def get(key):
    entry = cache.get_with_metadata(key)
    if entry and entry.expires_at - now() < random.uniform(0, 30):
        # Probabilistically refresh before expiry
        refresh_async(key)
    return entry.value if entry else None

Each request has a small chance of triggering an early refresh. The refresh load is spread evenly over time.

Request coalescing

Group all in-flight requests for the same key into one backend call.

9. Cache poisoning and security

A cache that returns malicious data is worse than no cache. Two attacks to defend against:

Cache poisoning via response splitting

# Attacker controls a query parameter that becomes part of the cache key
# and the response. They construct a parameter that produces a response
# containing a JS payload, then any user who hits that cache key gets the
# payload.

Defenses: never trust user input in cache keys; sanitize response headers; use signed cache keys.

Cache deception

# Attacker tricks the user's browser into caching a private page under
# a public-looking URL. Defenses: set Cache-Control: private on all
# authenticated responses.

10. Sizing and capacity planning

Cache sizing is a function of three things:

Working set size — how much data your workload actually touches.
Acceptable miss rate — what hit ratio you need for the cache to be worth it.
Memory cost — how much you are willing to spend.

The empirical rule: a cache holding 10% of the working set typically gets 80%+ hit ratio. Beyond 30%, the marginal hit-ratio gain shrinks dramatically. The optimum is workload-dependent; measure.

12. Key takeaways

Caching is on the critical path. Treat it like a hot loop.
Pick the eviction policy to match the workload (LRU for general, LFU for skewed, ARC for adaptive).
TTL bounds staleness; pick the smallest TTL your correctness allows.
Cache-aside is the default; reach for write-through or event-driven only when you need stronger consistency.
Watch for the thundering herd on hot keys.
Monitor hit ratio, not just latency. Hit ratio is the only metric that tells you whether the cache is doing its job.
Sizing: 10% of working set → 80%+ hit ratio is a reasonable starting point.

Appendix: terms

TTL (Time-To-Live) — the maximum age of a cached entry before it expires.
Hit ratio — the fraction of reads that find the value in cache.
LRU — Least Recently Used eviction policy.
LFU — Least Frequently Used eviction policy.
ARC — Adaptive Replacement Cache; hybrid of LRU and LFU.
Stampede / thundering herd — concurrent misses on the same hot key, overwhelming the backend.
Negative cache — caching the fact that an item does not exist, to avoid repeated queries for missing keys.

Appendix: source dialogue excerpt

The audio for this lesson was synthesized from the following Cantonese dialogue (verbatim, not translated):

M: 各位同學早晨, 我係子謙。歡迎收聽系統架構課程第一課。今日嘅主題係緩存, 英文叫 Caching。…
F: 大家好, 我係曉晴。緩存係系統架構入面最常見嘅性能武器, 但係用得唔好會變成最難 debug 嘅 source of bug。今日我哋會拆解緩存嘅原理, 同埋乜嘢時候應該用, 乜嘢時候應該唔好用。…
M: 首先講解基本概念。緩存嘅核心原理係 locality, 即係 temporal 同 spatial locality。Temporal locality 即係剛剛用過嘅數據好快會再用, 例如 loop 入面嘅變數。Spatial locality 即係附近位置嘅數據會一齊被用, 例如 array 連續讀取。緩存就係 e…
F: 緩存可以喺好多 layer 出現。CPU 有 L1 L2 L3 cache, OS 有 page cache, 瀏覽器有 HTTP cache, CDN 有 edge cache, 應用層有 Redis, 數據庫有 buffer pool。每個 layer 嘅 metric 都唔同, 但係 fundamental tr…
M: 好, 第一個重要概念係命中率。Hit ratio, 即係 cache hit 次數除以總 request 次數。如果 hit ratio 係百分之九十, 即係十個 request 入面有九個唔需要 hit 返原本嘅 source。Hit ratio 由零升到百分之九十嘅時候, average latency 可以跌百分…
F: 但係 hit ratio 唔係越高越好。從零到百分之八十, 大部分 workload 嘅 latency 都會顯著下降。但係由九十五到九十九, 你需要用更多 memory, 更複雜嘅 eviction policy, 結果 miss 嘅部分仍然係 latency tail, 你只係將 cost 推到 cache mis…

Full dialogue contains 28 segments; see script_raw.json in the source folder.

Caching

Transcript