InferenceMesh

A predictable gateway before a faster model

An inference API has to answer two different questions: which backend can serve a request, and whether the system has enough capacity to accept it. A fast model does not help if the gateway keeps admitting work that cannot finish within a useful time.

I built InferenceMesh around those decisions. Its current v0.1 implementation is a Python gateway with a deterministic local backend. It makes admission, routing, streaming, and API behavior inspectable without a GPU or a paid model endpoint.

What the implementation does

• Exposes chat completions, server-sent event streaming, embeddings, and model listing through an OpenAI-compatible HTTP interface.
• Bounds global concurrency, concurrency per tenant, queue depth, and the time spent waiting for capacity.
• Filters backends by task, health, circuit state, and streaming support before applying a deterministic weighted score.
• Returns request IDs and route-decision headers so a caller can inspect the backend, model version, policy version, and reason codes.
• Includes optional API-key authentication, request limits, health endpoints, Prometheus metrics, contract tests, and container and Kubernetes deployment definitions.

Request path

The HTTP layer validates the request and the requested model, then reserves capacity before beginning a response. The service layer selects a route and calls a backend through a small adapter contract. That separation keeps provider-specific request shapes out of the API contract.

HTTP request
  -> authentication and validation
  -> bounded admission
  -> eligibility filters and route scoring
  -> deterministic backend adapter
  -> JSON response or event stream

Request IDs, route reasons, and metrics accompany the path.

The router first removes candidates that cannot do the work. It then scores the remaining candidates using latency, queue utilization, estimated cost, quality penalty, and recent errors. The current values are controlled inputs for exercising routing policy; they are not measurements from a live fleet of GPU workers.

Three decisions that shape the project

A queue is a limited resource. Both active work and waiting work need bounds. A capacity reservation has one release path, and admission happens before streaming headers are sent. This makes overload an explicit API outcome instead of a request that waits indefinitely.

Route eligibility comes before optimization. A cheap backend that cannot serve the requested task is never a good route. Filtering before scoring makes this rule structural and gives route decisions a concrete explanation.

A deterministic backend is useful evidence. Repeatable text and embeddings let API tests inspect response shape, stream termination, authentication, unknown-model behavior, and queue exhaustion. The backend exists to verify gateway mechanics. It does not establish language-model quality or inference throughput.

How to inspect the behavior

The source includes separate admission, routing, authentication, chat, embedding, and health tests. A useful verification sequence is to submit a supported request, repeat it with streaming enabled, request an unsupported model, and then exhaust the configured admission limit. Those cases exercise different parts of the contract; a successful health check alone would miss most of them.

The repository also includes a smoke script and benchmark scaffolding. Benchmark claims require a pinned model, hardware, workload, configuration, and raw results; no GPU performance result is claimed here.

Current boundary and next experiment

The implemented release uses the local adapter. A gRPC contract is present, while a running gRPC transport, vLLM generation, Triton embeddings, shared caching, and rollout automation are later milestones.

The next useful experiment is a real backend adapter with cancellation and failure tests, followed by a controlled comparison between direct-backend latency and gateway latency. That would show the overhead of these controls under a defined workload.

Read the engineering note: inference gateways under load