LLM GPU VRAM & Hardware Sizing Calculator
Estimate exact GPU VRAM for Model Weights, KV Cache at high context, and multi-GPU cluster hardware.
1. Model Architecture & Quantization
2. Inference Context & Concurrency
8,192 tokens (8k)
8k
32k
64k
128k
1 stream
Inference Ready
GB VRAM
Weights:
KV Cache:
Overhead:
Recommended GPU Hardware Configurations
How to Calculate LLM GPU VRAM Requirements for Inference
Running open-source large language models (such as DeepSeek-R1, Meta Llama 3.3 70B, Qwen 2.5, or Mistral) locally or in cloud production requires accurately budgeting GPU memory. Running out of VRAM leads directly to CUDA out of memory crashes or severe CPU offloading bottlenecks.
Total inference VRAM is governed by three independent components:
Total VRAM (GB) = Model Weights Memory + KV Cache Memory + CUDA Runtime Overhead1. Model Weights Memory Formula
The base memory required just to hold the neural network parameters in VRAM depends solely on the parameter count and the quantization bit-depth:
- 16-bit (FP16 / BF16):
Parameters (Billions) × 2.0 Bytes. Example: Llama-3-70B requires ~140 GB. - 8-bit (FP8 / Q8_0 GGUF):
Parameters (Billions) × 1.0 Bytes. Example: 70B requires ~70 GB. - 4-bit (AWQ / GPTQ / Q4_K_M GGUF):
Parameters (Billions) × 0.55 Bytes(accounting for scale/zero-point metadata). Example: 70B requires ~38.5 GB. - 2-bit (Q2_K / EXL2):
Parameters (Billions) × 0.35 Bytes. Example: 70B requires ~24.5 GB.
2. KV Cache Scaling & Attention Mechanisms
As the conversation or document length grows, the attention Key-Value (KV) cache grows linearly with context tokens and batch size:
KV Cache Bytes = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element × context_length × batch_sizeModern models utilize Grouped-Query Attention (GQA) (such as Llama 3 with 8 KV heads vs 64 Query heads), reducing the KV cache memory footprint by 8x compared to legacy Multi-Head Attention (MHA). Furthermore, DeepSeek-V3 and R1 utilize Multi-Head Latent Attention (MLA), which compresses key-value vectors into a low-dimensional latent space (576 dimensions), allowing extreme 128k context scaling with minimal VRAM overhead.
3. Reference VRAM Table for Popular Open-Source LLMs
| Model | Parameters | 4-bit (Q4_K_M) | 8-bit (FP8) | 16-bit (FP16) | Recommended GPU |
|---|---|---|---|---|---|
| Llama 3.1 8B | 8.03B | 5.8 GB | 9.6 GB | 17.5 GB | 1x RTX 3060 12GB |
| Qwen 2.5 14B | 14.7B | 9.8 GB | 16.5 GB | 31.2 GB | 1x RTX 4080 16GB |
| Llama 3.3 70B | 70.6B | 41.5 GB | 76.0 GB | 148.0 GB | 2x RTX 3090 (48GB) |
| DeepSeek-R1 | 671B MoE | 380.0 GB | 710.0 GB | 1,380 GB | 8x H100 80GB (640GB) |
Frequently Asked Questions (FAQ)
Can I run Llama 3.3 70B on a single 24GB RTX 4090?
A 70B model quantized to standard 4-bit (Q4_K_M) requires approximately 39–41 GB of VRAM, which does not fit on a single 24GB card. However, you can either: (1) Run extreme 2.2-bit EXL2 quantization (fitting in ~22 GB with limited quality), (2) Pair two RTX 3090/4090 GPUs for 48GB combined VRAM, or (3) Use CPU RAM offloading with llama.cpp / Ollama, which allows running the model slowly over system RAM.
Why does context length cause CUDA Out of Memory during generation?
When generating responses for long documents (e.g. 32k to 128k tokens), the Key-Value (KV) cache stores past token activations in VRAM. For a 70B model with FP16 KV cache, a 64k token context consumes an additional 18+ GB of VRAM solely for cached tokens. Using FP8 KV Cache (--kv-cache-dtype fp8 in vLLM) cuts this cache footprint in half with zero perceptible loss in generation coherence.
