A night of CPU inference work, including everything that failed

Working session on a 6-core laptop, no GPU, 7.6 GB RAM, against Llama-3.2-3B.

What worked

• int8 with per-column scales. 2.82 GB, resident. Cosine 0.999949, tokens

unchanged through all 28 layers.

• Batching. 3.4-8x, throughput only — single-stream latency is untouched.
• Lloyd-Max quantiser levels. Placing levels on the observed density rather

than uniformly cuts error 35% at four bits, 52% at three. Costs build time and nothing at inference.

• KV prefix reuse. A follow-up turn reusing 11 of 12 cached positions: 9x.

What failed

Ternary, magnitude pruning, contextual sparsity, low-rank subspace projection, content-addressed deduplication, operation fusion, and multi-core threading. Each is written up separately. Deduplication returned exactly 1.000x at every chunk size from 64 bytes to 16 KB — quantised weights measure 7.025 bits per byte of entropy, and zlib reaches the same floor.

The retraction

An earlier draft of this work reported "68 s to 0.7 s per token." That is not a referenceable measurement. The 68 s baseline was an unoptimised numpy path re-reading 6.4 GB from disk every token — our own worst case, not anyone's starting point.

Against a real baseline on the same machine, CPU-only:

We are 2.1x slower than llama.cpp. The work produced a correct and resident int8 CPU engine, not a fast one, and the value is in the equivalence proofs and the negative results.

The thing that decided everything

Arithmetic intensity at one token is 1.78 FLOP per byte — memory-bound by roughly six times. Fusing 197 matrix-vector products down to 113 moves zero bytes and changed nothing. Only fewer bytes per weight, or more work per byte, ever moved the number.