2 posts tagged with "gpu-inference"

Most teams serving LLMs in production are burning money on GPU capacity they don't need. The root cause isn't carelessness — it's that GPU memory sizing for LLM inference involves four interacting variables (model weights, KV cache, activation memory, and framework overhead), and getting any one wrong means you over-provision the entire stack. When you multiply that error across multiple models on shared infrastructure, the waste compounds fast.

The math itself isn't hard. But most teams never do it, because "just give it an 80GB A100" is easier than calculating whether a 48GB L40S would suffice. This article walks through the arithmetic that determines how many models you can pack onto a single GPU — and the quantization tradeoffs that make it possible.

Most teams running LLM inference treat GPU provisioning like a guessing game. They see a model needs "140 GB at FP16," panic, requisition four A100-80GB cards, and call it done. What they don't calculate is how KV cache, concurrency, and quantization interact to determine the actual memory footprint — and that miscalculation typically means they're paying 3x more than necessary.

The math isn't complicated. But almost nobody does it before signing the cloud contract. This article walks through the exact formulas, shows where the hidden memory sinks live, and explains the bin-packing strategies that let you serve four models on hardware budgeted for one.