Massive activations invalidate naive rank measurements
A methods note rather than a result, and the kind of thing that quietly ruins an analysis.
Llama-3 hidden states contain a small number of channels running 52-83x the median magnitude. They dominate variance completely, so any rank or compressibility measurement taken naively reports near-rank-1.
| layer | max/median | effective rank, raw | after removing 8 channels |
|---|---|---|---|
| 0 | 17x | 1.9 | 2.1 |
| 7 | 83x | 1.0 | 19.2 |
| 14 | 62x | 1.0 | 58.0 |
| 21 | 52x | 1.1 | 15.0 |
| 27 | 16x | 2.7 | 3.8 |
An order of magnitude, from excluding 8 channels out of 3072.
What it means in practice
If you are measuring whether activations occupy a low-dimensional subspace — to justify a projection, a low-rank cache, or a compression scheme — strip the massive-activation channels first and report both numbers. The raw figure is not wrong so much as it is a measurement of those few channels rather than of the representation.
Reproduce with subspace_rank.py, which reports raw and corrected side by side.