Inference Servers
GPUs, batching, and engines.
A plain web service works for small models. For large neural networks, especially on graphics cards, specialised inference servers do much better. They keep the model in memory, batch requests automatically, and squeeze every bit of speed out of the hardware. Let's meet the main ideas, and the main choices.
- Specialised servers for big models
- Automatic batching
- Squeeze the hardware
Popular inference servers
There are several well known servers. Triton, from Nvidia, serves many model formats, with dynamic batching, on graphics cards and processors. Torch Serve is made for Py Torch models. Tensor Flow Serving does the same for Tensor Flow. K Serve, which we met, provides the Kubernetes layer around them. For large language models, specialised engines like vLLM use clever memory tricks, to serve many users at once, from one graphics card. Choose based on your framework, your hardware, and how much throughput you need.
| Server | Strength |
|---|---|
| Triton | many formats, dynamic batching, GPU or CPU |
| TorchServe | PyTorch-native |
| TensorFlow Serving | TensorFlow-native |
| KServe | Kubernetes layer: autoscaling, canaries |
| vLLM and similar | high-throughput language model serving |
What an inference server does
Inside, an inference server does four jobs. It loads models into memory, and keeps them warm, so there is no start up delay. It receives requests, and groups them into batches, within a tiny time window. It runs each batch on the graphics card, as one efficient operation. And it reports metrics, queue length, batch sizes, latency, and graphics card use, which you feed into monitoring and autoscaling. Because a graphics card is expensive, keeping it busy, is the main goal.
- Keep models warmno start-up delay
- Group requestsdynamic batching
- Run on the GPUone efficient operation
- Export metricsqueue, batch size, latency, GPU use
Metrics that matter on GPUs
Watch these numbers. Graphics card utilisation, if it is low, you are paying for idle hardware, perhaps because batching is too small. Memory use, models and batches must fit, and running out crashes the server. Queue length and wait time, which show if you need more replicas. Latency percentiles, as always. And cost per thousand predictions, the number that ties engineering to the budget. Autoscale on queue length, or concurrency, rather than only on processor use.
Summary
Excellent. Inference servers keep models warm, batch requests, and run them efficiently on graphics cards. Triton, Torch Serve, K Serve, and v L L M are common choices. Watch utilisation, memory, queues, and cost per prediction. Next, the opposite extreme, running models on phones and small devices.
Key takeaways- Warm models, dynamic batching
- Pick by framework and hardware
- Watch GPU utilisation and cost