Model Serving

Training produces a file of weights. Serving turns that file into something an app can call. A company that wants to run an open-weights model on its own GPUs starts a serving engine, points it at the weights, and gets an HTTP endpoint that accepts chat requests, often in the same format as a commercial API so existing code works unchanged.

Most of a serving engine’s work is fitting many requests onto expensive hardware. Continuous batching lets new requests join while others are still generating. Careful KV cache management, which vLLM’s PagedAttention introduced, leaves room for more requests at once. Prefix reuse (prompt caching), quantization and speculative decoding cut cost and delay further. The trade-offs show up as throughput against latency and time to first token.

“Inference engine” is a common name for the same software, but older AI used the phrase for the reasoning component of an expert system, so check which one a text means. Serving can also mean simpler models: a fraud classifier behind a REST endpoint is served too, usually with far less hardware.

In episode 72 at 3:03, AMD’s Anush Elangovan describes asking an agent to serve a model with vLLM on AMD’s ROCm stack: the agent finds the weights, reads the documentation, sets up everything needed for vLLM serving and keeps tuning performance afterward. That is his account of AMD’s own tooling. Episode 72 names Svix and Walrus Memory as presenting sponsors in its opening read.

See inference for what happens inside a single request, and batch inference for jobs that don’t need an always-on endpoint.