A timeout is an unknown outcome, not a failed operation

A worker requests a shipment reservation. Five seconds later, its HTTP client times out. Should the worker retry?

The exception does not contain enough information to decide. The remote system may never have received the request. It may also have created the reservation and lost the response. Retrying those two histories can have different consequences even though the caller observed the same error.

Reliable recovery starts by representing that uncertainty directly.

Two histories, one observation

History A: request never arrives
worker -> network failure                  -> timeout

History B: response never arrives
worker -> reservation created -> response lost -> timeout

In history A, another attempt may be necessary. In history B, a fresh operation could reserve a second shipment.

HTTP defines idempotency in terms of the intended effect of repeated identical requests. It also limits when clients should automatically retry non-idempotent methods. The method name alone does not supply an idempotency contract for an application-specific action. RFC 9110, idempotent methods

Use explicit operation states:

Unknown should retain the operation identity, original request, last observation, and time of dispatch. That gives a later worker enough evidence to continue recovery after a crash.

Retry the same intent

An idempotency key identifies one logical operation across transport attempts. Generate it before the first dispatch and persist it with the operation. A retry sends the same key and the same intent. A later, intentionally separate reservation receives a new key even if its fields happen to be identical.

The server needs a contract for concurrent duplicate requests, incompatible payloads under the same key, in-progress operations, and retention. AWS’s description of idempotent APIs explains why client request identifiers are useful and why identical request parameters do not always mean identical intent. Amazon Builders’ Library

A fingerprint of normalized request fields can detect accidental key reuse with a changed payload. It supplements the key; it does not decide whether two business actions were intended to be the same action.

For an effect contained in one database, the deduplication record and effect can be committed in one transaction. For a remote side effect, recording a key locally cannot make the remote action atomic. The destination must honor the identity, or the caller needs reconciliation with a trustworthy lookup mechanism.

Reconciliation has to answer the right question

A useful status API looks up the original operation identity. Searching for a similar reservation by customer and timestamp may find an unrelated operation, miss an eventually consistent write, or confuse an amended request with its predecessor.

Even a not found result needs documented semantics. A lookup can return nothing while the original request is still in flight. Absence is strong evidence only if the destination’s protocol establishes that the original attempt can no longer commit, or retries remain safe under its idempotency contract.

This pure decision function illustrates the distinction. replay_safe means the destination guarantees deduplication for this persisted key within the current retention window. absence_final means the lookup proves the original attempt cannot later create the effect.

def next_action(observation, *, replay_safe, absence_final=False):
    if observation == "succeeded":
        return "record-success"
    if observation == "failed-terminal":
        return "record-failure"
    if observation == "in-progress":
        return "wait-and-query"
    if observation == "not-found":
        if replay_safe or absence_final:
            return "retry-original-intent"
        return "reconcile-or-escalate"
    if observation == "unavailable":
        return "retry-original-intent" if replay_safe else "wait-and-query"
    raise ValueError("Unrecognized observation")

This chooses a recovery action; a scheduler still needs deadlines, retry limits, backoff with jitter, and a route to an operator when uncertainty persists. Never let wait-and-query silently become an infinite loop.

Current intent can change during recovery

Suppose the customer changes the delivery address while reservation status is unknown. The next worker now has two responsibilities: establish what happened to the old reservation and respect the current request revision.

Do not overwrite the payload associated with the old idempotency key. That destroys the meaning of replay. Keep the original operation immutable and create a separately reviewed operation for the amendment when appropriate.

Before dispatching a recovery action, claim it conditionally against the current workflow revision. The claim should bind the worker, operation ID, expected revision, and intended action. If the version has changed, reassess the action from the new state.

A database claim does not stop an already-dispatched request from completing later. Where required, the destination must also understand fencing tokens or conditional versions. Otherwise the workflow must detect late completion and reconcile it explicitly.

Compensation is another operation

Canceling a reservation is a new side effect with its own identity, preconditions, and uncertain-outcome window. It is not an automatic inverse that restores history.

For example, an original reservation might complete after a cancellation attempt reported not found. The workflow can then appear canceled locally while an active remote reservation exists. Recovery must account for both attempts and their eventual outcomes.

This is why a timeout should not immediately trigger an unrelated compensating call. First establish what is known, which operations may still complete, and which actions the destination can safely deduplicate or condition on a version.

Test histories, not just exceptions

A deterministic simulator can produce the same caller-visible timeout from both histories while retaining ground truth about remote effects. The recovery code sees only the allowed observations; the evaluator checks the actual state.

Useful scenarios include:

• A timeout before dispatch and a timeout after commit, with identical caller error messages.
• Two recovery workers claiming the same operation.
• A status lookup that temporarily returns not found before a late commit becomes visible.
• An amendment that invalidates a previously selected recovery action.
• A process crash after remote success but before the local success record is stored.
• A retry after the destination’s idempotency retention window has expired.

Assert invariants such as no duplicate reservation for one intent, no silent use of a stale revision, and no success record without confirmation. Measure how long workflows remain unknown as well as how many eventually finish. Completion rate alone cannot tell whether recovery respected the intended action.