An inference gateway needs a queue limit and a stream owner

A model endpoint can become unavailable while its model server remains healthy. The gateway accepts requests faster than the backend can complete them, callers wait, and retries add more waiting work. Memory grows before the backend reaches an obvious failure state.

An inference gateway therefore needs two contracts: how much work it will accept, and who owns that work until the response ends. Streaming makes the second contract especially easy to get wrong.

InferenceMesh explores this boundary with a deterministic backend. Its current gateway includes bounded admission and a lease held through streaming. The examples below explain the design problem; they do not represent measured GPU performance or a claim that every failure case is already implemented.

Limit waiting work as well as running work

Consider an illustrative service with eight backend slots. If 1,000 requests arrive together, a semaphore can keep active execution at eight while 992 coroutines remain alive waiting for a permit. The execution limit worked. The memory and latency boundary is still missing.

Python’s semaphore blocks acquisition when its counter reaches zero; it does not impose a separate limit on how many callers may wait. Its asyncio primitives also belong to an event loop and are not thread-safe. Python synchronization documentation

Use separate bounds for separate resources:

A queue absorbs a short mismatch between arrivals and completions. It cannot create capacity. Increasing its size often trades visible rejections for requests that expire after consuming resources.

For an interactive endpoint, immediate rejection can be a reasonable initial policy. If a queue is necessary, reserve a queue position atomically, release it on every exit, and use a deadline that includes time already spent waiting. A request should not receive a fresh end-to-end budget at every layer.

The response must own the reservation

This lifecycle is tempting:

request handler acquires a slot
request handler constructs a streaming response
request handler releases the slot
framework starts consuming the stream

It limits response-object construction. Generation is still running after the reservation has disappeared.

Ownership must follow execution instead:

validate -> acquire -> start response -> consume stream -> close -> release
                       |                    |
                       +------- failure ----+

The following transport-independent sketch makes that ownership explicit. capacity.acquire is assumed to enforce bounded admission and return an idempotently releasable lease. backend.open_stream owns cancellation and closure of the upstream request, including cleanup when its calling task is canceled. The transport methods are illustrative interfaces, not a framework API.

async def serve_stream(request, transport, backend, capacity):
    lease = await capacity.acquire(request.tenant)
    try:
        async with backend.open_stream(request) as source:
            await transport.start(status=200)
            async for chunk in source:
                await transport.send(chunk)
            await transport.finish_successfully()
    finally:
        lease.release()  # Synchronous and idempotent in this example.

The try begins immediately after acquisition, before any other awaited operation. That matters: instrumentation or response setup can fail too. Cleanup must cover the entire period in which the lease exists.

In a real ASGI integration, inspect when the framework starts the response and how it cancels response tasks. If releasing a lease needs asynchronous I/O, use the framework’s supported cancellation-safe cleanup mechanism with a bounded cleanup deadline. A finally block alone does not make arbitrary awaited cleanup immune to cancellation.

Closing the HTTP connection also does not prove that a remote model server stopped generating. Distinguish the local response reservation from backend work that is still draining. If backend cancellation cannot be confirmed, track that work separately and let backend admission remain authoritative; immediately treating its execution capacity as free can exceed the intended limit.

An in-process gate also limits only one process. Four workers with eight local slots can admit 32 active requests. Enforce backend-wide capacity through deliberate partitioning, backend admission, or shared coordination; do not label a process-local counter a global service limit.

Headers and tokens create different boundaries

There are two milestones to observe.

After response headers start, the gateway can no longer replace the response with an ordinary HTTP error status. An upstream failure before the first token may already be too late for a clean 503 response. Some gateways prefetch an initial upstream event before starting the downstream response. That adds buffering and changes the first-byte path, but gives early failures a normal HTTP representation.

After output reaches the caller, transparently switching providers can change the meaning of the answer. A second model has its own generation state. Appending its output to a partial answer can duplicate text or produce a contradictory continuation.

Server-sent events supply an event framing format, including event names and data fields. They do not define an inference application’s success marker or retry semantics. An application-specific error event is useful only when the client understands it. MDN’s SSE format reference

Define a client contract for incomplete streams. EOF without the expected completion signal must not automatically mean success. Record whether failure happened before headers, before content, or after content; these are operationally different outcomes.

Fairness is a scheduling problem

A per-tenant semaphore prevents one tenant from actively using every slot. Acquisition order can still undermine fairness. If requests acquire a global permit and then wait for a busy tenant permit, that tenant’s waiters can occupy global capacity while doing no backend work.

One alternative is a scheduler with bounded per-tenant queues that selects eligible work before assigning a global slot. That improves control over fairness at the cost of more state and scheduling logic. Keep the first design small, then use a mixed-tenant workload to determine whether it provides the intended isolation.

Tests that establish the contract

These checks belong in a real-server test, because an in-memory HTTP client may buffer responses or handle disconnects differently:

• Hold all execution slots, fill the waiting queue, and verify the next request is rejected without growing the queue.
• Cancel a waiter and confirm both queue depth and partially acquired permits recover.
• Disconnect after the first content event and verify upstream generation stops and the lease returns.
• Fail the backend before headers and after several events; assert the client sees the documented outcome in each case.
• Repeatedly saturate one tenant while another sends small requests; measure whether the second tenant can progress.

Track admission wait, time to first content, stream duration, rejections, and cancellations separately. A fast first token does not establish a healthy completion path. The useful invariant is that accepted work remains accounted for until it finishes or has been stopped.