LoRA in Practice

A big language model has billions of weights. Updating them all takes serious hardware. LoRA's insight is that you don't need to. Freeze everything, and train a tiny add-on, that nudges the model towards your task.

• Billions of weights
• Freeze them all
• Train a tiny add-on

An adapter that learns

Here's the idea, on a toy layer. The base layer is frozen. A and B are two tiny matrices, and their product adds a small correction to the output. Only they get trained. Run it. The loss falls, even though not one original weight changed. Real tools, such as Unsloth and PEFT, apply this across a whole model. The adapter's file may be only a few megabytes, and you can swap adapters for different jobs.

import torch
from torch import nn

base = nn.Linear(16, 16)
for p in base.parameters():
    p.requires_grad = False

rank = 2
a = nn.Parameter(torch.randn(rank, 16) * 0.01)
b = nn.Parameter(torch.zeros(16, rank))

target = torch.rand(16, 16)
x = torch.eye(16)
opt = torch.optim.Adam([a, b], lr=0.05)

for step in range(201):
    y = base(x) + x @ a.T @ b.T
    loss = ((y - target) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (0, 100, 200):
        print(f"step {step:>3}  loss {loss.item():.4f}")

print("Base weights trained: 0  Adapter weights:", a.numel() + b.numel())

step   0  loss 0.3311
step 100  loss 0.0689
step 200  loss 0.0680
Base weights trained: 0  Adapter weights: 64

Measuring generated text

How do you score a generated reply? Old metrics, like BLEU, count overlapping words with a reference answer. We try a simple version on four sentences. Run it. A perfect copy gets one. But look at the third. It's complete nonsense, with the words shuffled, and it scores a perfect one. And the second, a great short reply, scores lower. Word overlap can't judge meaning. That's why we now prefer human review, or a model judge, for generated text.

from collections import Counter

def overlap(candidate, reference):
    c = Counter(candidate.split())
    r = Counter(reference.split())
    return sum((c & r).values()) / max(1, sum(c.values()))

reference = "your order will arrive on monday"

for text in ["your order will arrive on monday",
             "your order arrives monday",
             "monday on arrive will order your",
             "we sell many fine books"]:
    print(f"{overlap(text, reference):.2f}  {text}")

1.00  your order will arrive on monday
0.75  your order arrives monday
1.00  monday on arrive will order your
0.00  we sell many fine books

Summary