The stale cache write that happens after invalidation

Cache invalidation often begins with a reasonable rule: when a database row changes, delete its cached value. The next request will load the new row.

That rule assumes the next cache write starts after the deletion. An older read may still be in flight.

The four-step race

Imagine a cached configuration document with a revision number:

Reader A                         Writer B
--------                         --------
cache miss
read revision 41 from database
                                 commit revision 42
                                 invalidate cache entry
write revision 41 into cache

The invalidation was delivered successfully. The cache still ends up stale.

A TTL limits how long an entry survives after insertion; it does not stop this insertion. If the old read finishes late, its TTL begins late too. More reliable event delivery alone also cannot determine whether an in-flight fill is older than a completed invalidation.

The cache needs enough history to reject a result that has already been superseded.

Keep a version floor after deletion

Associate each logical key with a monotonic revision. An invalidation for revision 42 means: future entries for this key must be at least revision 42. Retain that version floor even when the cached payload is absent.

For a single process, this small model demonstrates the required atomic transitions. Values are strings to avoid mutable payloads escaping the lock.

from threading import Lock


class VersionedCache:
    def __init__(self):
        self._lock = Lock()
        self._floor = {}
        self._entries = {}

    def invalidate(self, key: str, revision: int) -> None:
        with self._lock:
            floor = max(self._floor.get(key, 0), revision)
            self._floor[key] = floor
            current = self._entries.get(key)
            if current is not None and current[0] < floor:
                del self._entries[key]

    def fill(self, key: str, revision: int, value: str) -> bool:
        with self._lock:
            if revision < self._floor.get(key, 0):
                return False
            self._floor[key] = revision
            self._entries[key] = (revision, value)
            return True

    def get(self, key: str):
        with self._lock:
            return self._entries.get(key)

The fill also advances the floor. That prevents two concurrent cache misses from moving the value backward even when no invalidation event has arrived.

This model assumes positive integer revisions are strictly ordered for a key and the same revision always denotes the same immutable content. Validate that contract at ingestion. It omits TTLs, eviction, capacity limits, and distributed storage. Those are additional design decisions, not properties supplied by the lock.

Make the race deterministic in a test

Tests should force the relevant ordering instead of hoping a load test reproduces it. Once reader A holds revision 41, pause it. Deliver invalidation 42, then allow A’s fill to continue.

The model’s transition assertions are compact:

cache = VersionedCache()
cache.invalidate("routing", 42)
assert cache.fill("routing", 41, "old") is False
assert cache.get("routing") is None

assert cache.fill("routing", 42, "new") is True
cache.invalidate("routing", 40)  # A delayed event cannot move time back.
assert cache.get("routing") == (42, "new")

assert cache.fill("routing", 43, "newer") is True
assert cache.fill("routing", 42, "new") is False

Also test duplicate invalidations, deletion followed by recreation, and two fills completing in reverse order. If a recreated record resets its revision to one, an old high version may dominate forever. Use a persistent revision sequence or an incarnation identifier with an explicit ordering scheme.

Distribution adds two different problems

First, the comparison and write must remain atomic. Reading a version from Redis and then issuing a separate unconditional SET recreates a race between the check and the write. Redis documents optimistic locking with WATCH, where the transaction fails if a watched key changes before execution. A server-side atomic operation is another option. Redis transaction documentation

If the store shards keys, co-locate the relevant state or use an operation supported across the required keys. Treat aborted transactions as retries with a bounded budget, and include eviction behavior in the design.

Second, the event must become visible. A version floor prevents regression after a consumer has observed that version. It does not make an asynchronously replicated cache immediately consistent with the database. A read can still return revision 41 while the event for revision 42 is delayed.

An outbox written in the same database transaction as the state change can make event publication recoverable. It still needs a publisher, consumer progress tracking, deduplication, and reconciliation. It does not remove publication lag.

State the actual read contract. Examples include eventual freshness, bounded freshness subject to a measured lag budget, or reads that carry a required minimum revision and bypass the cache when it cannot meet it. If a caller must see its own write, returning the committed revision gives its next read a useful minimum-version token.

A missing entry must not erase history

The version floor is sometimes called a tombstone or watermark. It cannot be discarded casually while older work can still complete. If an expired floor disappears and an old reader returns, that reader may populate the key again.

A fixed retention interval works only if there is a justified upper bound on old reads, message delivery, and replay. Otherwise use durable version metadata, an authoritative revision check, or a recovery protocol that reestablishes floors before accepting fills. A consumer restart should have a defined warm-up procedure; replaying old events into an empty process can temporarily permit older state.

Versioned immutable cache keys offer another tradeoff: store routing:42 and keep the current-revision pointer separately. Old fills then populate old keys without replacing the new payload. The pointer still needs trustworthy publication and monotonic updates, and readers need a clear rule for obtaining it.

Measure correctness symptoms alongside hit rate

A high hit rate can coexist with a high rate of incorrect hits. Observe invalidation lag, rejected stale fills, revision mismatches, consumer progress, and fallback reads. These signals show whether the freshness contract is being met.

The useful invariant is precise: once a cache has observed revision 42, it must never publish revision 41 for that key again. Achieving that invariant requires retaining ordering information through deletion, eviction, and restart.