The same model, 76% faster, same chip

We did not swap the chip or the model. We changed how the model is stored, and it got much faster while taking far less room.

The test we locked

One model, one desk box, one fixed question. We served the same 8B model three ways and sent each the exact same prompt with greedy decoding and a 200-word ceiling.

The first way is the convenient default: one command, an endpoint on a port, and you are running. The second way compiles that same precision into a tuned engine first. The third way switches to a 4-bit mode, where each weight is packed down to four bits and the chip runs that math in hardware instead of unpacking it first.

We measured the same two things every time: raw typing speed in words a second, and how much memory the model held while it answered.

What happened

The convenient default ran at 22.1 words a second. The compiled default, same precision, nudged it to 24.6. The 4-bit mode ran the same model at 38.8 words a second. That is 76% faster than the convenient default, on the same chip, with no change to the job the model does.

Here is the full board.

The 4-bit mode also held about 2.5 GB while it ran, against 11.2 GB for the convenient default. That is roughly a quarter of the memory for the same answers, which on a single shared box is room for a search index or a second model to sit beside it.

The honest part

We will not oversell the middle row. Compiling the same precision into a tuned engine only bought about 10 to 15% over the convenient default. That alone was not worth the extra setup. The real jump came from the 4-bit mode, and it came for a specific reason: this chip runs 4-bit math in hardware. On a chip without that, software 4-bit does not give you this win, so the result is tied to the hardware, not a free lunch everywhere.

A 4-bit mode also trims some precision to win the speed and the memory. For the work we run it on the trade held and the answers stayed good, but that is a thing to check on your own task, not assume. The point we stand behind is narrow and real: on this box, the same model went 76% faster and held a quarter of the memory, by changing only how it is stored.

This ran on Orionfold Arena, the local-AI cockpit, on a desk box you can own. See Arena.

Why this can be trusted

We did not change the model or the chip, so the speed cannot be explained by a better model or better hardware. The only thing that varied across the three runs is how the weights are stored, which is exactly the lever the claim is about. We served all three the same prompt with the same decode settings, so the comparison is apples to apples.

We also published the weak result next to the strong one. The compiled-default row barely moved, and we showed it rather than hiding it to make the 4-bit win look bigger. And we named the reason the win exists, which is the chip’s hardware 4-bit path, so you can tell whether your own box will see the same jump before you expect it.

Rerun it

Serve the same 8B model on the same desk box three ways: the convenient default, a compiled default at the same precision, and the 4-bit mode the chip runs in hardware. Send all three the same prompt, time the words a second, and read the memory each one holds while it answers. If your numbers do not match ours, tell us.