InferenceMesh

The problem

Model-serving systems need to balance an application-friendly API with operational choices that are anything but simple: model selection, batching, latency, GPU utilization, version changes, and degraded-provider behavior.

Current release

The first verified release establishes the gateway foundation before introducing GPU cost and model variability. It currently includes:

• OpenAI-compatible chat completions, SSE token streaming, embeddings, and model listing.
• Bounded global, per-tenant, and queue admission with explicit overload responses.
• Hard backend eligibility filters followed by deterministic weighted routing and decision headers.
• Optional API-key authentication, input/output limits, request IDs, health checks, and Prometheus metrics.
• A deterministic local backend that makes the complete control flow reproducible without pretending to be a GPU benchmark.
• Automated tests, Docker and Kubernetes assets, CI, benchmark scaffolding, and a versioned gRPC contract.

Target architecture

The backend adapter boundary is designed for vLLM text generation and Triton embeddings or reranking. Later milestones add Redis-scoped caching, MLflow champion/candidate aliases, Kafka batch inference, evaluation gates, and automatic rollback. Ray Serve remains deferred until a measured multi-stage pipeline gives it a distinct responsibility.

request -> validation -> bounded admission -> route policy -> backend adapter
                                                   |              |
                                                   |              -> vLLM / Triton
                                                   -> metrics + reason codes

Engineering decisions

• Bounded concurrency and queueing prevent a burst from exhausting process or backend memory.
• Streaming preserves responsiveness without pretending the whole response is available.
• Capability and health constraints are applied before performance scoring.
• Model and provider choices remain visible in response metadata and telemetry.
• Performance claims will be published only with model, hardware, workload, configuration, and raw-result evidence.

Next milestone

The next release integrates a real vLLM backend and Triton gRPC embeddings, adds circuit-breaking and cancellation verification, and measures direct-backend versus gateway overhead on pinned reference hardware.