Convert model parameters to GPU VRAM

Use a quick calculation to reject impossible hardware choices before paying for an instance.

Base formulas

Examples

A 13B model is roughly 26 GB in FP16, 13 GB in INT8 and at least 6.5 GB at 4-bit before overhead. A 70B model is roughly 140 GB in FP16, 70 GB in INT8 and at least 35 GB at 4-bit. This explains why a quantized 70B model may fit on a 48 GB GPU while the FP16 version needs multiple large GPUs.

Why file size can disagree

Repositories may contain several formats, optimizer states, duplicate shards or adapters. Total repository size is not necessarily the memory loaded for one inference configuration. Inspect the exact files selected by the runtime.

Recommended margin

For a first test, target a GPU with at least 20% more memory than the calculated weights. Increase the margin for long-context generation, batching, image/video pipelines or unfamiliar runtimes.