Developer Tools
LLM Inference VRAM Calculator
Estimate the minimum GPU VRAM an LLM needs to run, for any parameter count and precision
B
Common models
Quantization
—GB
Model weights (minimum reference)Model weights—
Inference overhead (~20%)—
Recommended total VRAM—
Mainstream GPU VRAM reference
The figure here is a minimum for inference — it leaves out training, which usually needs 10–20× more. Real usage also depends on batch size, sequence length and KV cache, so treat this as an estimate.
How it works
GPU memory use comes down to three things:
- Parameter count: 7B means roughly 7 billion float parameters — Llama 3.1, for one — and each one takes as many bytes as the precision asks for.
- Quantization: FP32 costs 4 bytes per parameter, FP16/BF16 2, INT8 1 and INT4 0.5. Lower precision costs less memory, but quality drops with it.
- Inference overhead: at runtime you also hold activations, the KV cache and the framework itself, which usually adds another 10%–30%.
The formula: weights (GB) = parameters (B) × bytes per parameter (decimal GB, 1 GB = 10⁹ bytes).
Bytes per parameter by precision
PrecisionBit widthBytes / parameter7B model
FP3232-bit4 B28 GB
FP16 / BF1616-bit2 B14 GB
INT88-bit1 B7 GB
INT44-bit0.5 B3.5 GB
Training VRAM
Training also has to hold gradients and optimizer state — Adam keeps two momentum terms — so it usually needs 12–20× what inference does. This tool covers inference only; training is out of scope.