LoRA changes how a weight update is represented. Instead of training a full replacement for each weight matrix, it learns a low-rank update while the pretrained weights stay frozen. Hu and colleagues tested this approach on several language models and reported comparable or better quality than full fine-tuning in those experiments (Hu et al., LoRA). The paper’s results are tied to its models and tasks; they are not a guarantee for a new workload.
A concrete use is a support-ticket classifier with an adapter trained on your organization’s labeled tickets. Keep a held-out set containing confusing labels, rare categories and tickets that need escalation. Compare the adapted model with the base model under the same prompt. Store the base checkpoint, adapter version and label definitions together so a later result can be reproduced.
Fergal Reid describes Intercom’s early experiments with parameter-efficient fine-tuning methods in episode 49 (11:25).
For builders, the misconception to avoid is that fewer trainable parameters make data quality or evaluation optional. Start with a clear task and measure whether the adapter changes the behavior you need. Fine-tuning explains the broader process; quantization is a separate reduction in numerical precision. Can an open model cut your AI costs? follows the Intercom case and the operating work around training.
Sources
- Hu et al., LoRA — Primary source for the explanation and distinctions in this entry.
Go deeper
- PEFT: LoRA docs
Work through rank, target-module selection and adapter initialization before choosing a LoRA configuration.
- QLoRA: Efficient Finetuning of Quantized LLMs paper
Study how a frozen quantized base and trainable low-rank adapters work together in the QLoRA method.
