Adapters and LoRA

The Idea

Full fine-tuning rewrites every weight, which means a full copy of the model per task. A low-rank adapter instead learns two thin matrices whose product has the same shape as the layer, and adds it. The base weights never move, so one base can serve many tasks.

Real-World Example

A tailor does not reweave a suit. They take in the waist and shorten the sleeves — a handful of stitched corrections, reversible, and a different set for each wearer.

The Code

d, r = 4096, 8                  # layer width, adapter rank
full = d * d                    # a full fine-tune rewrites all of these
adapter = 2 * d * r             # B is d×r, A is r×d
print(full, adapter)            # 16777216 65536
print(round(100 * adapter / full, 2), "%")   # 0.39 %
# merged weight = W + B @ A -> same shape, so serving cost is unchanged

d, r = 1024, 4
adapter = ___ * d * r
print(adapter)   # 8192