VRAM Calculator For LLM

LLM VRAM Usage Estimator

Model Size (Billion Parameters):

Quantization / Precision: FP16 / BF16 (16 bits) INT8 (8 bits, e.g., Q8_0) ~6 bits (e.g., Q6_K) ~5.5 bits (e.g., Q5_K_M) ~5 bits (e.g., Q5_0, Q5_K_S) ~4.5 bits (e.g., Q4_K_M) ~4 bits (e.g., Q4_0, Q4_K_S, NF4) ~3.5 bits (e.g., Q3_K_M) ~3 bits (e.g., Q3_K_S) ExLlamaV2

ExLlamaV2 BPW (Bits Per Weight):

Context Length (Tokens):

Estimated VRAM usage will be displayed here.

Calculation Method & Disclaimer:

• This calculator provides a rough estimate. Actual VRAM usage can vary.
• Calculation: VRAM ≈ Model Weights + KV Cache + Fixed Overhead.
• Model Weights: `Model Size(B) * Bits Per Weight / 8` GB.
• KV Cache (Approx.): Uses a simplified formula based on size, context, and quantization level. This is a very rough part of the estimate.
• Fixed Overhead (Approx.): Assumed ~1.0-1.5 GB for software, CUDA, etc.
• Bit numbers next to quant formats are approximate averages.
• Real-world usage depends heavily on loader software (llama.cpp, ExLlamav2, vLLM, etc.), batch size, drivers, and specific model implementation.
• Treat this as a guideline, not an exact figure.