Architecture notes
Engineering, explained.
Notes on backend systems and AI infrastructure: start with a failure, work through the design, and make the trade-offs explicit.
01
An inference gateway needs a queue limit and a stream owner
Why concurrency limits alone do not protect an inference API, and how response ownership changes once a token stream starts.
02
Keep policy decisions separate from LLM explanations
A practical boundary for AI-assisted workflows: versioned facts and deterministic decisions, with an explanation that cannot grant permission.
03
The stale cache write that happens after invalidation
Deleting a cache entry does not stop an older reader from putting it back. Monotonic versions make that race explicit and testable.
04
A timeout is an unknown outcome, not a failed operation
How idempotency, reconciliation, and version checks make recovery safer when a remote operation may have committed before the caller timed out.