Low-Rank Adaptation (LoRA) is a fine-tuning technique that adapts a large pre-trained model to a new task by training a small number of additional parameters, rather than updating the model's full, massive set of original weights. It works by inserting compact, low-rank matrices into specific layers of the network and training only those, while keeping the original pre-trained weights frozen and unchanged.
Because it only trains a small fraction of the parameters a full fine-tune would touch, LoRA is dramatically more memory-efficient and less computationally expensive, which lets teams adapt large models to specialized tasks using far less hardware than full fine-tuning would require. This efficiency has made LoRA one of the most widely used techniques for customizing large language models and image generation models alike, since it lowers the cost and technical barrier to producing a task-specific version of an existing foundation model.
