LoRA: Low-rank Adaptation

Many applications in natural language processing rely on multiple sub-adaptations of LLM models. To run models on more specialized tasks, there are options such as RAG architecture, few-shots, system prompting, etc. But if you want a more specialized model then you need to fine-tune it with your own data.

Fine-tuning has some disadvantages. These:

• In traditional fine-tuning, it is a big problem that the model contains as many parameters as the original model. This fine-tuning process allows the original weight matrix (W) of the network to be modified. Changes made to (W) during fine-tuning are updated in aggregate. The residual weights can be expressed as (W) + (Δ W).
• A lot of resources are spent during the tuning of millions of parameters.
• There is a risk of forgetting information from the original training. This is called catastrophic forgetting . It occurs when the model forgets previous knowledge acquired during pre-training. This results in incorrect output for the pre-trained information, as the fine-tuning affects the model’s weights.

BUT DON’T WORRY!

Minimizing these disadvantages and making LLMs work more efficiently is possible with the LoRA technique.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S.,Wang, L. & Chen, W. (2021). Lora: Low-rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685.

What is LoRA?