Chapter 4.7 β€” Parameter-Efficient Fine-Tuning: LoRA and QLoRA

Contents

  1. Why full fine-tuning stops being practical
  2. LoRA: learning a low-rank update instead of a full one
  3. Why a low-rank update is enough
  4. Where adapters go, and merging them back at inference
  5. QLoRA: quantizing the frozen base
  6. Memory arithmetic: what you actually save
  7. Limitations and when full fine-tuning still wins
  8. Interview angle
  9. Self-check questions
  10. Sources

1. Why full fine-tuning stops being practical

Chapters 4.5 and 4.6 described SFT, RLHF, DPO, and their alternatives as if updating "the model" during fine-tuning were free of any further complication beyond choosing the right objective. It isn't. Full fine-tuning means running gradient descent over every parameter of the pretrained model, and Chapter 4.4 already established what that costs: gradients the same size as the weights, and β€” for Adam, the optimizer essentially everyone uses β€” two more same-sized buffers for the running moment estimates. Add it up in mixed precision and full fine-tuning needs on the order of four times a model's raw parameter footprint just for weights, gradients, and optimizer state, before a single activation is stored. For a 7B-parameter model that's comfortably outside what a single consumer or even a single datacenter GPU holds; for anything past about 30B parameters it requires exactly the kind of multi-GPU sharding machinery β€” ZeRO, FSDP β€” that Chapter 4.4 covered for pretraining itself.

That cost would be easier to accept if every fine-tuning run needed to move every parameter by a meaningful amount. It doesn't have to be that way, and the rest of this chapter is about a family of methods that exploit that fact directly: freeze almost all of the pretrained weights, and learn only a small number of new parameters that capture whatever the downstream task actually requires. Done well, this shrinks the trainable parameter count by several orders of magnitude, which shrinks the optimizer-state and gradient memory by the same factor β€” the two things that dominate full fine-tuning's memory bill in the first place β€” while leaving the model's core capability, established by expensive pretraining, entirely untouched.

2. LoRA: learning a low-rank update instead of a full one

Low-Rank Adaptation, LoRA, introduced by Hu et al., asks a simple question about any weight matrix $W \in \mathbb{R}^{d \times k}$ that fine-tuning would otherwise update in place: instead of learning an arbitrary update $\Delta W$ of the same shape as $W$, what if we constrain $\Delta W$ to be low-rank? Concretely, LoRA freezes the pretrained $W$ entirely and represents the update as a product of two much smaller matrices, $$\Delta W = BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},$$ where the rank $r$ is chosen to be far smaller than either $d$ or $k$ β€” typically single digits to a few dozen, against hidden dimensions in the thousands. The forward pass through the adapted layer becomes $$h = Wx + \frac{\alpha}{r}\,BAx,$$ where $\alpha$ is a fixed scaling hyperparameter that lets you tune the adapter's effective learning rate independently of $r$. $A$ is initialized with small random values and $B$ is initialized to exactly zero, so that $\Delta W = 0$ at the very start of training β€” the adapted model is numerically identical to the frozen pretrained model before a single gradient step happens, and training then gradually opens up the low-rank update from nothing.

Only $B$ and $A$ receive gradients; $W$ never does. In the original paper's experiments, applying this to just the attention projection matrices ($W_q$ and $W_v$) of GPT-3 175B at rank 4 cut the number of trainable parameters by roughly four orders of magnitude relative to full fine-tuning, while matching or exceeding full fine-tuning's downstream task performance. That last part is the surprising piece: constraining the update to a tiny fraction of the full parameter space's dimensionality barely costs anything in quality, on the tasks and scales the paper measured.

3. Why a low-rank update is enough

The empirical justification for LoRA's central bet predates the paper itself. Aghajanyan et al.'s work on the intrinsic dimensionality of fine-tuning showed that the parameter update needed to adapt a pretrained language model to a specific downstream task lives in a surprisingly low-dimensional subspace β€” far lower-dimensional than the model's raw parameter count would suggest β€” and that this intrinsic dimensionality shrinks further as the base model gets larger. In other words, bigger pretrained models don't just get better; they get easier to specialize, because whatever needs to change to specialize them is confined to an increasingly small subspace of the full parameter space. LoRA's rank-$r$ decomposition is a direct, structural bet on that finding: if the useful update really does live near a low-dimensional subspace, a rank-$r$ matrix product can represent nearly all of it for a small $r$, and the parameters LoRA doesn't spend on capturing that subspace aren't wasted potential β€” they were never going to move far from zero in a useful direction anyway.

This explains why LoRA's quality holds up so well relative to full fine-tuning at surprisingly small ranks (often $r=4$ to $r=16$ across a wide range of tasks and model sizes) rather than needing $r$ to scale with the model's hidden dimension. It also predicts, correctly, that the effect gets more pronounced at larger model scale β€” exactly the direction Aghajanyan et al.'s result points in, and exactly what LoRA's own ablations on progressively larger models confirm.

4. Where adapters go, and merging them back at inference

LoRA can in principle be attached to any linear layer β€” attention projections, the feedforward layer's matrices, or both β€” and the original paper's own ablations found that spreading a fixed parameter budget across more of the attention projections (rather than concentrating it on just $W_q$ at a higher rank) tends to work better, suggesting the useful low-rank update is distributed across several projections rather than concentrated in one. In practice, the choice of which matrices to adapt and at what rank is one of the few hyperparameter surfaces this method leaves you with, and different downstream tasks can prefer different choices.

The detail worth being precise about in an interview is what happens at deployment. Because $\Delta W = BA$ has exactly the same shape as $W$, you can compute $W' = W + \frac{\alpha}{r}BA$ once, offline, and serve $W'$ directly β€” the deployed model is a single ordinary dense matrix per adapted layer, indistinguishable in structure and inference cost from a model that was fully fine-tuned. This is a genuine advantage over an earlier family of parameter-efficient methods, bottleneck adapters (Houlsby et al.), which insert a small extra feedforward block serially between existing layers: those adapters add a real forward-pass cost at inference time, on every request, forever, because they're a separate computation that has to run in sequence. LoRA's update is additive to an existing matrix rather than a new serial step, so merging removes the adapter as a runtime cost entirely. The tradeoff is that merging commits you to one adapter at a time per served copy of the weights β€” serving many different task-specific adapters against the same base model concurrently means either keeping them unmerged and paying a small batched-matrix-multiply cost per request, or keeping multiple merged copies of the full weights around, one per adapter.

5. QLoRA: quantizing the frozen base

LoRA already shrinks the optimizer-state and gradient memory dramatically by making almost every parameter frozen. QLoRA, introduced by Dettmers et al., asks the next question: if $W$ is frozen and never receives a gradient, does it need to sit in memory at 16-bit precision at all? QLoRA's answer is to quantize the entire frozen base model to 4 bits, using a purpose-built data type called NF4 (4-bit NormalFloat) whose quantization levels are chosen to be information-theoretically optimal for weights that are approximately normally distributed β€” concretely, the 16 representable levels are placed at the quantiles of a standard normal distribution, so that each level covers an equal share of probability mass rather than an equal slice of the numeric range β€” which pretrained transformer weights empirically are β€” rather than using a generic uniform quantization scheme that wastes precision on values far from where the actual weight distribution has mass.

Two further tricks make this practical rather than just theoretically appealing. Double quantization applies a second round of quantization to the quantization constants themselves (the per-block scale factors that any block-wise quantization scheme needs to store), shaving off additional memory that would otherwise be spent on bookkeeping rather than the weights themselves. Paged optimizers use NVIDIA's unified-memory paging to move optimizer state between GPU and CPU memory automatically during the rare, unpredictable spikes that gradient checkpointing (recomputing rather than storing certain activations, trading compute for memory β€” see Chapter 4.4) can cause, avoiding an out-of-memory crash instead of requiring you to provision permanent headroom for a worst case that occurs on a small fraction of steps. During the forward and backward pass, the 4-bit weights are dequantized back to a computable precision on the fly, layer by layer, so the arithmetic itself still happens at reasonable precision β€” the 4 bits is a storage format, not a computation format.

The result reported in the QLoRA paper is striking on its own terms: fine-tuning a 65B-parameter model on a single 48GB GPU, matching the performance of full 16-bit fine-tuning on the tasks they measured, at a scale that would otherwise require dozens of high-end GPUs just to hold the model. That combination β€” a frozen, quantized base plus small full-precision LoRA adapters riding on top β€” is now the default recipe for fine-tuning large open-weight models on hardware far smaller than the base model would otherwise demand.

6. Memory arithmetic: what you actually save

It's worth working through concrete numbers rather than leaving the savings as a qualitative claim. Take a 7B-parameter model. Full fine-tuning in mixed precision needs roughly: 2 bytes per parameter for the fp16 weights, 2 bytes per parameter for fp16 gradients, and β€” for Adam β€” two fp32 optimizer moment buffers plus (commonly) an fp32 master weight copy, another 12 bytes per parameter. That's on the order of 16 bytes per parameter, or around 112GB for 7B parameters, before counting a single activation β€” well outside a single 24GB or even 80GB GPU without sharding across multiple devices.

LoRA changes the picture entirely because the 7B frozen base weights need no gradients and no optimizer state at all β€” they need only sit in memory at inference precision, roughly 14GB in fp16. The trainable LoRA parameters, at a rank low enough to attach to every attention projection across a 7B model's layers, typically number in the low tens of millions rather than billions β€” often well under 1% of the base model's parameter count. Those tens of millions of parameters are the only ones needing gradients and Adam state, at roughly the same 16-bytes-per-parameter rate, adding well under a gigabyte. The total lands around 15GB rather than 112GB: comfortably inside a single consumer GPU.

QLoRA cuts the dominant remaining cost, the frozen base weights, by roughly 4x again: 4-bit storage means the same 7B-parameter base occupies around 3.5–4GB instead of 14GB, with the small LoRA adapters contributing the same negligible amount as before. Scaled up to the paper's headline case, a 65B model that would need on the order of a terabyte of memory for full 16-bit fine-tuning fits, quantized, in under 48GB β€” the entire reason single-GPU fine-tuning of models that size became routine practice rather than a research curiosity.

7. Limitations and when full fine-tuning still wins

The low-rank assumption is an empirical bet, not a guarantee, and it can fail to hold as cleanly for tasks that genuinely require the model to acquire new knowledge or behavior far from anything in its pretraining distribution, rather than reweighting and recombining capabilities it already has. In those cases, the gap between LoRA and full fine-tuning can widen, and the usual response β€” raising the rank $r$, or adapting more of the network's weight matrices β€” starts eating back into the very parameter-efficiency that motivated using LoRA in the first place, without a guarantee of fully closing the gap.

Serving many task-specific adapters concurrently against a shared base model is a genuine strength of the unmerged form (many small $(B, A)$ pairs instead of many full model copies) but it does mean the deployment story is more involved than "run the model" β€” it requires infrastructure that selects and applies the right adapter per request, a system-design concern Chapter 6.1 picks up directly. And QLoRA's clean merge trick from LoRA doesn't carry over quite as neatly: merging $BA$ into a base matrix that's stored in 4-bit NF4 isn't a simple addition the way it is for a 16-bit base, so a served QLoRA model typically either dequantizes the base back to a mergeable precision (giving back some of the memory savings at serving time) or keeps the adapter unmerged and applies it alongside the quantized base at inference, paying a small runtime cost that plain LoRA's merge trick was specifically designed to avoid.

Taken together with the supervised fine-tuning, preference optimization, and alignment techniques covered in Chapters 4.5 and 4.6, parameter-efficient fine-tuning closes out this part's account of how a pretrained model turns into something useful: trained to follow instructions, aligned to behave acceptably under pressure, and now cheaply specializable to whatever new task or domain you actually need it for, without paying full fine-tuning's cost every time. What none of that touches is a different axis entirely β€” what a model with a fixed set of weights does at inference time, and whether it can be made to think longer and more carefully about a given problem before committing to an answer, rather than simply emitting its best first guess. That question is where Part V begins, with Chapter 5.1, "Reasoning and Test-Time Compute."

8. Interview angle

A senior LLM interview will rarely ask you to derive LoRA's math from scratch, but it will probe whether you understand exactly where the savings come from and what they cost, usually through questions like:

  • "Why does freezing the base weights save memory, when a very small learning rate on all parameters would also produce a small update?" β€” a strong answer identifies that the savings come from removing gradients and optimizer state for the frozen parameters entirely, not from the update being numerically small; a tiny learning rate applied to all parameters still requires storing a gradient and Adam state for every one of them.
  • "How does LoRA avoid adding inference latency, when adapter-based fine-tuning methods in general have a reputation for doing exactly that?" β€” a strong answer distinguishes LoRA's additive, mergeable update ($W' = W + \frac{\alpha}{r}BA$, same shape as $W$) from serially-inserted bottleneck adapters (Houlsby et al.), which run as an extra forward-pass step at every inference call and cannot be merged away.
  • "What's actually being assumed when you pick a rank $r$, and why does it tend to work even at very small values?" β€” a strong answer connects this to the intrinsic-dimensionality result predating LoRA: the useful fine-tuning update for a given task is believed to live in a low-dimensional subspace of the full parameter space, and that subspace shrinks further as the base model grows, which is why small, roughly model-size-independent ranks tend to suffice.
  • "What does QLoRA add on top of LoRA, and what specific problem does each piece solve?" β€” a strong answer separates the three pieces cleanly: 4-bit NF4 quantization of the frozen base cuts the dominant remaining memory cost; double quantization shaves the bookkeeping overhead of the quantization constants themselves; and paged optimizers absorb memory spikes from gradient checkpointing without requiring permanent headroom provisioned for the worst case.

The through-line interviewers are checking for is whether you can trace memory and compute costs back to their actual source (frozen vs. trainable parameters, merged vs. serial computation, storage vs. compute precision) rather than treating "LoRA is more efficient" as an opaque fact to be cited.

9. Self-check questions

  1. Why does making almost all parameters frozen β€” rather than making the update to every parameter small β€” produce most of LoRA's memory savings?
  2. What is the intrinsic-dimensionality finding that motivates constraining $\Delta W$ to be low-rank, and how does it change with model scale?
  3. Why can a LoRA adapter be merged into the base weight matrix with zero added inference cost, while a bottleneck adapter (Houlsby-style) cannot?
  4. What does the NF4 data type do differently from a generic 4-bit quantization scheme, and why does that matter for pretrained weights specifically?
  5. What is a paged optimizer, and what specific failure does it prevent during QLoRA fine-tuning?
  6. Describe a fine-tuning scenario where you'd expect LoRA to underperform full fine-tuning, and explain why in terms of the low-rank assumption.

10. Sources

  • Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685. https://arxiv.org/abs/2106.09685
  • Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. arXiv:2305.14314. https://arxiv.org/abs/2305.14314
  • Aghajanyan, A., Zettlemoyer, L., & Gupta, S. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv:2012.13255. https://arxiv.org/abs/2012.13255
  • Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019. arXiv:1902.00751. https://arxiv.org/abs/1902.00751

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

What does LoRA freeze during fine-tuning?

Explanation: LoRA never updates $W$ itself; the only parameters receiving gradients are the much smaller $B$ and $A$ matrices whose product approximates the needed update.

Why is $B$ initialized to zero while $A$ gets small random values?

Explanation: With B = 0, Ξ”W = BA = 0 at initialization, so training starts from exactly the pretrained model's behavior and gradually opens up the low-rank update.

If $d = k = 4096$ and rank $r = 16$, how many trainable parameters does the $BA$ decomposition for one weight matrix have? (parameters = $2 \cdot d \cdot r$)

Explanation: 2 Β· 4096 Β· 16 = 131,072 β€” versus 4096 Γ— 4096 = 16,777,216 for the full Ξ”W, about 128Γ— fewer.

According to the intrinsic-dimensionality finding that motivates LoRA, what happens as the base model gets larger?

Explanation: Aghajanyan et al. found intrinsic dimensionality shrinks as models scale up, which is exactly why small, roughly model-size-independent LoRA ranks tend to suffice.

Why can a LoRA adapter be merged into the base weight matrix with zero added inference cost, while a Houlsby-style bottleneck adapter cannot?

Explanation: LoRA's update is additive to an existing matrix, so W' = W + (Ξ±/r)BA can be precomputed once; a serially-inserted adapter is extra computation that can't be folded away.

What does the NF4 data type optimize for, compared to a generic 4-bit quantization scheme?

Explanation: NF4's quantization levels are placed to match a normal distribution, which is empirically how pretrained transformer weights are distributed β€” a generic uniform scheme wastes precision elsewhere.

What specific problem do paged optimizers solve in QLoRA?

Explanation: Paged optimizers use unified-memory paging to move optimizer state to CPU RAM during unpredictable spikes, rather than requiring permanent GPU headroom for a worst case.

Full fine-tuning of a 7-billion-parameter model needs roughly 16 bytes per parameter total (fp16 weights + fp16 gradients + fp32 Adam state and master weights). Approximately how many GB is that?

Explanation: 7e9 parameters Γ— 16 bytes β‰ˆ 112GB β€” well outside a single 24GB or 80GB GPU without sharding.

What's a genuine limitation of LoRA relative to full fine-tuning?

Explanation: The low-rank assumption is an empirical bet, not a guarantee β€” it holds less well when the task needs behavior far outside what the pretrained model already has some grip on.

Why doesn't QLoRA's adapter merge as cleanly into the base weights as plain LoRA's does?

Explanation: A quantized base's values are discrete, so a served QLoRA model typically either dequantizes to merge (giving back some memory savings) or keeps the adapter unmerged at a small runtime cost.