An int8 dot-product kernel takes the engine to 5.3 tokens per second on four ARM cores, output unchanged
On a four-core ARM CPU (Neoverse-N1, Oracle Cloud), running Hermes-3 (Llama-3.2-3B) with int8 weights in a single stream, two changes took the engine from 2.89 to 5.3 tokens per second, end to end in its live configuration:
| change | tok/s | what stays the same |
|---|---|---|
| before | 2.89 | — |
| vectorised associative-memory lookup | 3.54 | bit-identical: 6,844 of 6,844 results compared, before and after new material is written into the memory |
int8 × int8 dot products (vdotq_s32), activations scaled per 16 values | 5.3 | greedy output identical for all 48 tokens on each of 6 prompts; perplexity 30.037 against 30.039 |
The first change only stopped the per-token memory lookup from walking ~85,000 entries in Python. The second is the one that matters: the weights were already int8, but each product widened them to floating point first. Multiplying int8 by int8 needs the activations in int8 too, and how they are scaled decides whether the model survives it:
| activation scaling | teacher-forced top-1 agreement | KL (nats) | greedy, 6 prompts × 48 tokens | model step |
|---|---|---|---|---|
| none: float activations (the exact path) | reference | — | reference | 194 ms |
| one scale per vector | 90.5% | 0.0397 | diverges after 1–37 tokens | 99 ms |
| one scale per 64 values | 97.8% | 0.0013 | one prompt diverges at token 21 | 105 ms |
| one scale per 32 values | 99.3% | 0.0007 | identical | 128 ms |
| one scale per 16 values | 99.2% | 0.0005 | identical | 110 ms |
One scale per vector fails for the reason in an earlier note: a few channels run tens of times larger than the rest and set the scale for everything. Small blocks keep those channels from swamping the others. Agreement was measured on the model alone, without the memory bias or sampling, over 768 teacher-forced positions; perplexity over 2,048 tokens of WikiText-103 and Project Gutenberg text. Step times are medians from one run, all under the same background load; the 32-value block came out slower than the 16-value one there, so 16 is what runs live.
A 3B-class model at reading speed — about five tokens per second — on four CPU cores and no GPU is the near-term bar for the tutor work, and this clears it on this one machine.
What this does not show:
- It is a rented cloud server, not hardware a family owns. The claim is about the engine on ordinary CPU cores, not about an owned device yet. These cores are the server sibling of the Cortex-A76 in a Raspberry Pi 5; whether a Pi 5, with less memory bandwidth, reaches the same number is untested.
- It is not faster than llama.cpp. Measured the next day on the same machine, llama.cpp is about 1.5× faster at equal precision; the head-to-head is below.
- The x86 laptop is still at 2.09 tok/s. The kernel uses an ARM instruction. The laptop's CPU reports
avx_vnni, the x86 equivalent, so porting it is the next concrete step, not yet done. - Batching does not get cheaper. This kernel's cost grows almost linearly with the number of rows multiplied together (8 rows cost 7.8 times one), so the speed-up is for a single stream.
Against llama.cpp on the same machine (added 2026-10-12)
Measured in one window on the same four cores, under the same background load, with four threads and 48 generated tokens, prompt excluded:
| engine | tok/s |
|---|---|
| llama.cpp, Q8_0 (about the same bytes per weight as our int8) | 10.5 |
| llama.cpp, Q4_K_M | 13.5 |
| this engine, model loop only (step, readout, next token) | 6.9–7.0 |
| this engine as served, with its memory and other extras | 5.4 |
llama.cpp is about 1.5× faster at equal precision. Both engines read about the same number of bytes per token: 3.6 GB here (2.82 GB of int8 layers and a 0.79 GB bf16 output head) against 3.4 GB for llama.cpp's Q8_0. So the gap is how fast those bytes are read, roughly 25 GB/s against 36. The likely costs are about 196 separate kernel calls per token, each quantising its own input, and the bf16 output head. That is the next work. Both engines produced the same output on every run; one run of our model loop hit a load spike (3.4 tok/s) and is left out here, but kept in the raw log.
llama.cpp was commit e937b17, built from source with default CPU options, running NousResearch's official Hermes-3-Llama-3.2-3B GGUFs.
Measured on Oracle Cloud's Ampere A1 (4 Neoverse-N1 cores, no GPU), 2026-10-11, with the engine's own decode clock; fidelity by comparing the exact and int8 paths on the same inputs. Kernel, engine patch, scripts, the head-to-head and raw outputs: research/int8-kernel; the engine it patches is bqsm-neuron.