The activation subspace is not low-dimensional
If hidden activations occupied a small subspace, the linear paths through the network — about 75% of the weights — could be projected into it and read at a fraction of the cost. A small probe suggested exactly that.
| probe | dims for 90% of variance | % of 3072 | ceiling min(T, D) |
|---|---|---|---|
| 41 tokens | 16-28 | 0.6% | 41 |
| 318 tokens | 161-183 | 5.2-6.0% | 318 |
The dimensionality grew with the sample. It did not converge to a small constant, so 16-28 was an artifact of measuring 41 tokens against a ceiling of 41 — not a property of the model.
At k≈200, a rank-k projection reads 8192 × 200 instead of 8192 × 3072, which drops the theoretical ceiling from 4x to about 1.6x. Net of the cost of projecting in and out, nothing remains.
The methodological point
The first measurement was flattering because the probe was smaller than the quantity being measured. The check that caught it is cheap: run it again at a different sample size and see whether the number moves. If it tracks the sample, it is measuring the probe.
← Back to the library