Understanding LoRA & QLoRA GPU Memory Requirements
Fine-tuning Large Language Models (LLMs) requires careful resource management. Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) drastically reduce memory usage by freezing the base model and training only a small set of adapter weights.
1. How to Use the Calculator
- Select a Base Model: Choose a preset model or define custom parameters (Hidden Dimension, Layers, and Total Params).
- Choose Quantization: Select 16-bit for standard LoRA, or 8-bit/4-bit for QLoRA. Lower precision directly shrinks base model VRAM footprint.
- Adjust LoRA Rank (r): The rank dictates the expressiveness of the adapter. Higher rank increases trainable parameters, optimizer state, and gradients.
- Target Modules: Select which attention and MLP projections to inject LoRA into. More modules yield better performance but consume more VRAM.
- Set Training Config: Batch size and sequence length exponentially impact activation memory during the forward/backward passes.
2. Mathematics and VRAM Formulas
The total GPU memory required is the sum of several distinct components:
- Base Model Weights: Parameter Count × Bytes per parameter. (e.g., 4-bit quantization = 0.5 bytes per param).
- LoRA Adapter Weights: Trainable Parameters × 2 bytes (FP16/BF16).
- Optimizer States: Using AdamW requires tracking momentum and variance, using 8 bytes per trainable parameter.
- Gradients: 4 bytes (FP32) per trainable parameter to store gradients during the backward pass.
- Activations: Scales linearly with
Batch Size × Sequence Length × Hidden Dimension × Layers. Gradient checkpointing can reduce this at the cost of computation speed.
3. Key Features of this Tool
- Instant dynamic calculations entirely in your browser.
- Support for modern standard models (Llama 3, Mistral, Phi-3).
- Granular module targeting (q_proj, v_proj, gate_proj, etc).
- Automated GPU hardware recommendations based on final VRAM overhead (including a 1GB CUDA context buffer).
4. Reference Memory Table (QLoRA 4-bit)
| Model | Rank (r) | Target Modules | Est. Total VRAM |
|---|---|---|---|
| Llama 3 8B | 8 | q, v | ~6.5 GB |
| Llama 3 8B | 16 | All | ~7.8 GB |
| Llama 3 70B | 16 | q, v | ~39.5 GB |
| Llama 3 70B | 64 | All | ~44.2 GB |
5. Frequently Asked Questions
Why does sequence length consume so much memory?
Transformers calculate self-attention across the entire sequence. The memory footprint for storing intermediate activations during the forward pass scales linearly (and quadratically for vanilla attention without Flash Attention) with sequence length. Use Gradient Checkpointing to heavily reduce this.
Can I run QLoRA on a 12GB RTX 3060?
Yes. For 7B or 8B parameter models, 4-bit QLoRA with rank 16 and a batch size of 1 typically requires around 7.5 to 8.5 GB of VRAM, easily fitting inside a 12GB GPU.
Should I target all modules or just Q and V?
Recent research indicates that targeting all linear layers (Q, K, V, O, Gate, Up, Down) yields results closer to full fine-tuning. However, this increases trainable parameters and VRAM. If memory is tight, falling back to just Query and Value projections is standard practice.