Fine-Tuning Models (Part 3): QLoRA & 4-bit Quantization
A hands-on guide to fine-tuning LLMs with QLoRA using NormalFloat4 (NF4) quantization and Double Quantization on consumer GPUs.
QLoRA (Quantized Low-Rank Adaptation) brings the power of fine-tuning multi-billion parameter LLMs directly onto single consumer-grade GPUs (such as RTX 3090/4090).
Breakthroughs in QLoRA#
QLoRA introduces 3 major technical innovations:
- NF4 (NormalFloat4): An information-theoretically optimal 4-bit data type for normally distributed weights.
- Double Quantization: Quantizes the quantization constants themselves, saving ~0.37 bits per parameter.
- Paged Optimizers: Manages memory spikes by paging GPU memory over to CPU RAM dynamically.
Hands-On Setup with BitsAndBytes#
qlora_setup.py
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
# [!code focus]
# 4-bit NF4 Quantization Config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
quantization_config=bnb_config,
device_map="auto"
)Summary & Recommendations#
Fine-tuning is no longer limited to tech giants with massive compute clusters. Leveraging QLoRA, PEFT, and BitsAndBytes, you can train tailored models for custom domain tasks right on local GPU hardware.