Keep policy decisions separate from LLM explanations

Suppose a change-review tool checks whether a deployment has a rollback plan. The evidence says it does not. A language model produces a confident explanation that the change is low risk and can proceed.

If the application extracts its decision from that paragraph, prose has become an authorization mechanism. The system has no stable answer to a basic debugging question: which rule permitted this action?

A more inspectable design gives facts, decisions, explanations, and execution separate responsibilities. This article uses an invented deployment-review example to make that boundary concrete.

Start with a decision that has a shape

The policy engine consumes validated facts and produces a structured record:

{
  "decisionId": "review-104",
  "outcome": "deny",
  "reasonCodes": ["ROLLBACK_PLAN_MISSING"],
  "factsVersion": 17,
  "policyVersion": "change-policy-3"
}

The identifier is illustrative. The important fields are the outcome, reasons, and the versions that explain where they came from.

Reason codes should be part of a maintained interface. A UI can localize them, a test can assert them, and an audit query can count them without parsing natural language. Preserve structured evidence references separately so a reviewer can inspect the relevant fact.

Open Policy Agent describes a similar architectural separation: software asks for a policy decision and then enforces that decision at the application boundary. The decision engine does not itself provide every enforcement integration. OPA integration documentation

Determinism requires explicit inputs

A pure function is a useful starting point. Here the facts have already passed schema validation, including strict boolean types; None represents evidence that could not be established.

from dataclasses import dataclass


@dataclass(frozen=True)
class Facts:
    version: int
    tests_passed: bool | None
    rollback_ready: bool | None


@dataclass(frozen=True)
class Decision:
    outcome: str
    reasons: tuple[str, ...]
    facts_version: int
    policy_version: str


def evaluate(facts: Facts, policy_version: str) -> Decision:
    denied = []
    if facts.tests_passed is False:
        denied.append("TESTS_FAILED")
    if facts.rollback_ready is False:
        denied.append("ROLLBACK_PLAN_MISSING")

    if denied:
        outcome, reasons = "deny", tuple(denied)
    elif facts.tests_passed is None or facts.rollback_ready is None:
        outcome, reasons = "needs-review", ("EVIDENCE_INCOMPLETE",)
    else:
        outcome, reasons = "allow", ()

    return Decision(outcome, reasons, facts.version, policy_version)

This example defines a precedence rule: an established blocker produces a denial even when another fact is unknown. With no known blocker, missing evidence requires review. Different applications may choose a different policy, but the precedence needs to be intentional and testable.

The dataclasses document the internal contract; Python type hints do not validate untrusted JSON at runtime. The ingestion boundary still has to reject strings such as "false", invalid versions, and unexpected fields.

For reproducible evaluation, pin the policy artifact and reference data as well as the facts. If a rule depends on time, pass the relevant evaluation time as an input. If it depends on an external lookup, record the lookup result and its freshness. A version string with mutable contents behind it is insufficient for replay.

Let the model explain a stored decision

The execution path should be clear:

validated facts -> policy evaluation -> stored decision -> enforcement
                                            |
                                            +-> optional explanation

The explanation receives the stored decision and the minimum relevant evidence. It can answer: What blocked the change? Which check should the developer revisit? Which facts are missing?

It cannot rewrite outcome, remove reason codes, issue an approval, or invoke the executor. Those restrictions should follow from API permissions and data flow. A prompt asking the model to respect the decision is helpful presentation guidance, but it is not the enforcement boundary.

Store the generated explanation separately, with the decision ID it describes. If generation fails, the structured reasons are enough to render a useful fallback. If generated text contradicts the result, the UI must still show the authoritative outcome, and the application can discard the text.

Free-form rationales supplied by users are untrusted evidence. Including one in the explanation context must not allow instructions inside it to cross into policy evaluation or tool authorization. Sanitization, bounded inputs, and limited model capabilities support this separation.

A correct decision can still become stale

Now consider this sequence:

1. Revision 17 passes review.
2. Someone edits the change, producing revision 18.
3. An executor reads the old approval and applies revision 18.

The policy was correct when evaluated. The action used its result for a different object.

Bind approval to the reviewed revision and enforce that binding when claiming the work. A conditional state transition might look like this:

UPDATE change_request
SET state = 'executing', claimed_by = :worker_id
WHERE id = :change_id
  AND revision = :approved_revision
  AND state = 'approved'
RETURNING id, revision;

No returned row means the claim failed. Do not follow it with an unconditional update. In PostgreSQL’s Read Committed mode, a conflicting updater waits and then rechecks the condition against the updated row. PostgreSQL transaction isolation

This claim protects a database transition. An external deployment still needs an immutable revision-specific payload, an idempotent operation identity, and a recovery path if the process crashes after dispatch. Once execution is claimed, either prevent edits to that revision or make later edits a separate revision with its own review.

Evaluate both paths independently

Policy tests should cover missing evidence, every blocker, combined blockers, version changes, and repeated evaluation of identical snapshots. Include a stale-approval race and a pair of workers trying to claim the same revision.

Explanation tests ask different questions: Does the text preserve the outcome? Does it cite supplied evidence? Can it tolerate an unavailable model? Does an adversarial rationale change anything outside the explanation field?

The policy may become more complex as requirements grow. The explanation may become more useful as models improve. Keeping their responsibilities explicit lets either evolve without quietly changing who is allowed to act.