We Ran Transformers on Four Retro Consoles and Counted Every Cycle

The interesting part is that we can tell you exactly how many cycles each one takes, and we can tell you which of our own conclusions turned out to be wrong.

Every number comes from a real ROM running under an accurate emulator. Where a figure is an extrapolation rather than a measurement, we say so.

Three things that transfer

Low-bit weights win on memory traffic, not on arithmetic

Ternary weights (−1, 0, +1) measured 41% faster than int8 on the Nintendo 64's MIPS core, and 12% slower on the same console's RSP vector unit.

Both figures are sparsity-matched. Our shipped ternary model is 32% zeros and our ternary kernel skips them where the int8 kernel does not — the raw kernel-pair comparison reads 57%. Remove the zero skip from the ternary kernel and re-measure the same weights on the same ROM and it is 41%; add a zero skip to the int8 kernel instead and it is 49%. The missing multiply is worth 71% of the gap, the skipped zeros the other 29%. The RSP figure needs no such correction: a vector lane cannot skip a zero.

That is not a contradiction, and we tested both candidate explanations rather than picking one. The density advantage was real: ternary moved 3.74× fewer weight bytes. The unit was not bandwidth-bound either, running at roughly a fifth of RDRAM throughput. The cost was unpacking — 0.656 issue slots per weight against int8's 0.375.

On a vector unit whose vmadh performs eight 16×16 multiply-accumulates in a single instruction, there is no multiply to delete, so you pay only the unpack. On the 6502 the same question resolved a third way: a 4-bit weight with a lookup table is quick to read, but building the table index requires the accumulator, and the 6502 has no way to build it elsewhere. In the weight matmul, where both operands vary per element, the 4-bit design has to spill the accumulator to memory: measured at +6 cycles per element, 20 against 14. Ternary keeps the accumulator in A, and it does not store its zeros at all — at our model's density it costs 5.0 cycles per weight against the 4-bit design's 16.5 with a matched zero test, a 3.3× gap that survives matching the sparsity.

We had been calling that three different constraints — instruction set on the MIPS core, issue slots on the RSP, register pressure on the 6502. Then we measured a fourth machine and found we had been wrong about all of them.

The rule underneath: operand traffic, not arithmetic

The Super Nintendo has a signed hardware multiply in its picture processor: 16×8 into 24 bits, available with no wait states. On a console whose CPU runs at 3.58 MHz that should be decisive, and we predicted int8 would finally beat ternary there.

It does not. Measured before any engine was written, in exact cycles:

Ternary is 2.02× cheaper, and the multiply is not why. The PPU multiply is genuinely free — every one of the 214 cycles in that arm is data movement.

That is the whole mechanism, and it replaces our three explanations with one. Ternary's advantage is not that it deletes a multiply. It is that it halves the number of operands the machine has to fetch — and it therefore lasts exactly as long as the machine charges per operand:

It decays monotonically as machines get better at moving data, and it flips on the one unit that fetches eight operands in a single transaction. The RSP was never an exception to a rule about multipliers; it was the end of a gradient about memory.

This also makes a prediction we can be wrong about later: on any new target, ternary's margin should track memory transactions per accumulate, and should not care how good the multiplier is.

When you quantize matters more than how far

Post-training ternarization with round-to-nearest destroys our N64 model. The corpus is byte-level, so ln(256) = 5.545, and round-to-nearest with a BitNet-absmean quantizer lands above that in every seed we ran — worse than uniform random over bytes. But round-to-nearest is not the state of the art. Running GPTQ instead, on the same checkpoint and emitting the same 2,031,628-byte ternary blob, post-training quantization comes back to 0.3 to 1.4 depending on the seed. Training with the quantizer in the loop still wins — both arms sit at 0.101 ± 0.001 against a full-precision control at the same place — but by roughly 3–14×, not by 66×. Post-training quantization of this model is also far higher-variance: our QAT and full-precision arms move by 0.001 nats across seeds, our PTQ arms by one to two.

We checked whether quantization-aware training works by finding weights that happen to quantize well. It does not. The trained latent weights are 11.6% larger in mean magnitude than the float control's (0.0827 against 0.0741, across three seeds) — but that is the one statistic a mean-magnitude threshold cannot see: scale every weight by any constant and the ternary codes come out identical. The statistic that decides which weights survive barely moves: 42.7% zeros against 42.9%. The rest of the network absorbs the error during training instead.

The practical consequence is direct: quantization-aware training let a 2.80× larger model fit the same console memory at no measured difference in training loss, where the int8 version of that shape overruns it by 2.4×.

Build the instrument first, then distrust it

Every genuine gain here appeared only after a measurement tool existed — and every first tool was wrong in a way that silently reversed a conclusion.

• Static instruction counting swung from −17% to +1% on a no-op line reorder, and separately called a measured 14% improvement a regression.
• A three-token output check passed two genuinely broken changes. One deleted live range checks and diverged at position 5. The other narrowed cache entries to 16 bits while values peaked at 110,944 — overflowing 44 times inside the three-token window it passed.
• A benchmark ROM that omitted the interface, tokenizer and vocabulary could not see a memory-bank overflow. We submitted a pull request upstream that broke the build, and found it only when we finally linked the real ROM.
• One N64 emulator priced an uncached cartridge read at approximately zero cycles. An accurate one measures 268.6. The same optimization measures −7.47% on the first and −14.2% on the second — and the first is deterministic to the last digit across runs.
• Two of our own training comparisons were quoted to four decimals from a single seed. Re-run at three seeds, the difference in each was smaller than the spread of either arm, and two of the nine cross-seed pairings reversed the sign.
• Our Nintendo 64 token-rate counter computed tokens × 60 / loop-iterations while the loop generated exactly one token per iteration. The expression reduces to the constant 60. It reported 60.00 tok/s on two builds whose real speeds differ by 1.8×, and that number reached a repo README, a video description, this article and a company homepage before anyone tried to reproduce it. The honest figure is 1.22.
• A Genesis optimization measured −0.008%, and we nearly wrote it off. The build system's generated dependency files were never included, so changing the model weights did not invalidate the compiled object: the ROM contained the new engine and the old weights. Measured properly, the same change is −7.83%.

That last one is the variety worth fearing. A false positive gets caught, because someone eventually looks for the win and cannot find it. A false null is never investigated — it does not look like a bug, it looks like an idea that did not pan out. We had already declined this optimization once on a risk judgement. The second refusal would have been backed by a number, and the number would have been real, and it would have been measuring the wrong ROM.

The counterfactual

The Ricoh 2A03 shipped in 1975. Backpropagation was published in 1986. Cartridge bank switching, battery-backed memory, and every other primitive our NES port depends on were shipping consumer technology by 1985.

Our 6.36-million-parameter model represents on the order of 10¹⁵ training operations. A Cray-1 sustained roughly 160 MFLOPS in 1976, putting that at three to four months of machine time — expensive, and the kind of allocation researchers actually received.

The transformer architecture was published in 2017.

So the constraint was never the silicon. Small-scale neural inference was physically available for decades while the field pursued symbolic approaches: expert systems, LISP machines, hand-built ontologies. The hardware sat there the entire time.

A footnote on supercomputers

Full analysis: this section has its own page — Could a Cray Have Run GPT-2? When LLMs Became Buildable.

Consumer hardware is one budget; a national laboratory is another, and the distinction lands somewhere surprising.

A Cray-2 (1985) could be configured with up to 4 GB of memory. Fifteen hundred million 8-bit weights occupy about 1.5 GB, so they fit. Inference costs roughly two operations per parameter per token, which at that machine's ~1.9 GFLOPS works out to a second or two per token. A GPT-2-sized model could plausibly have been run — not trained — on mid-1980s supercomputing hardware.

Training is where it stops. GPT-2's training run is on the order of 10²¹ operations. At 1.9 GFLOPS that is roughly seventeen thousand years.

So the era could have executed a large model and could not have produced one — which makes the same point from the other end. The missing ingredient was never the machine. It was the architecture, and the compute to fill it.

When did it become buildable?

Full analysis: this section has its own page — Could a Cray Have Run GPT-2? When LLMs Became Buildable.

"Impossible" and "expensive" are different claims, and the boundary between them has a date. Below is the fastest computer on Earth in each year, against a single GPT-2 training run. The range reflects two credible estimates of that run's cost; the exact token count is debated.

Roughly 27 Cray-2 systems were ever manufactured. All of them, running flat out for a year, complete somewhere between 1.8% and 0.1% of one training run. The 1980s could not have done this for any amount of money — the hardware did not exist in sufficient quantity on Earth.

By 1993 that collapses from twenty-five thousand machine-years to about twenty. By 1999 a single existing machine does it in a year.

The machine nobody built

What follows is speculative engineering. Every component shipped, every figure is arithmetic on published specifications, and the machine itself never existed. We are labelling that boundary rather than blurring it.

Nobody had to wait for a single fast computer. The render farm was already the state of the art: Beowulf clustering of commodity hardware was published in 1994, and television 3D animation was being produced on banks of Amigas by 1992. Shard the work instead.

That design already existed. ASCI Red, delivered to Sandia in 1997, was 9,298 Pentium Pro 200 processors on a custom mesh. A GPT-2 run on it works out to 2.7 years.

The catch is not arithmetic, it is the network. Data-parallel training must all-reduce the entire gradient every step — 1.5 billion parameters at 32-bit is 6 GB. Over 10 Mbit links that is 9,600 seconds per step, and blades do not help: they reduce the number of slow links, not the bytes each one carries. A hundred thousand steps is thirty years of pure communication.

The lever is gradient accumulation, which is neither modern nor subtle — take fewer, larger steps.

At K = 100 the network cost falls to about four months, comfortably inside a two-year compute budget on 10 Mbit links.

It was expensive rather than impossible — a national-laboratory budget, not a miracle. Nobody built it, because the architecture it would have trained was not published until 2017.

Proof of Antiquity

Elyan Labs runs a blockchain that rewards vintage hardware for participating. Older silicon earns a higher multiplier, verified by hardware fingerprinting that emulators cannot fake. The premise is that old machines have worth beyond nostalgia.

This work is the other half of that argument. Not "old hardware deserves credit for existing", but "old hardware can do the thing everyone assumes requires a datacenter". A Game Boy Color generating a sentence is a stronger claim about antiquity than any multiplier we could assign it.

What we got wrong

We had two frontier models adversarially review these conclusions. They removed several, and the ones that survived are stated at the strength the evidence supports.

• None of the quantization techniques are new. Ternary weight networks, BitNet-style training, accumulator-aware training and cache quantization all predate this work. What is new is the measurement on these targets.
• Our "phase change" framing was confounded. Comparing 3-level to 8-level quantization changed the level count and the quantizer design together.
• Two platforms have now run on real hardware; two have not. A Sega Genesis Model 1 from 1988 generates text on a real console (video), and the rate on screen is the console counting its own vblanks rather than a figure we supplied. A Nintendo 64 runs the scalar build on real silicon via EverDrive 64 (video). That ROM's on-screen counter was broken and has been replaced. It computed tokens × 60 / loop-iterations, and the loop generated exactly one token per iteration, so it reduced to the literal constant 60 — it printed 60.00 on two builds whose real speeds differ by 1.8×. Re-measured against vertical blanks, the honest figures are 1.22 tok/s scalar and 2.16 with the vector unit. The RSP vector build has never been on silicon, and its speedup is 1.78×, not the 4.769× we published earlier — that figure was a cycle ratio on a weight format the cartridge does not ship. The Game Boy Color and NES ports are emulator-only. Two independent emulators agreeing is not silicon: our Game Boy assembly disables interrupts while the stack pointer walks an activation array, and a real console with a live display controller is less forgiving.
• We have withdrawn a number. An earlier version of this piece reported ternary as 53.6% faster than int8 on the Game Boy Color. That figure is an N64 CPU measurement that was misattributed; no Game Boy ternary kernel was ever built, so there is no Game Boy ternary speed measurement. The Game Boy's 10.09× in the table above is an int8-to-int8 optimisation stack and is unaffected.
• Our corpora are saturated. Every model we trained lands at essentially the same fit loss. We can show a 2.80× larger model fits; we cannot show it is better, because our data has no headroom left. That is a data problem and we are not pretending otherwise.

Reproduce it

All four ports, the measurement harnesses and the findings journals are public.

• Scottcjn/gbc-transformer — Game Boy Color, 10.09×
• Scottcjn/legend-of-elya-genesis — Sega Genesis, exact 68000 cycles
• Scottcjn/legend-of-elya-n64 — Nintendo 64, RSP vector matmul
• Scottcjn/elya-nes — NES / Famicom, 0.689 s/token, token-exact

The harnesses matter more than the ports. If you take one thing from this, take the habit: build the instrument, calibrate it against something you can derive by hand, and check it against a second implementation before trusting any number it gives you.