Spectra ships ternary 3.9B models that match 4-bit quantisation at half the bits

Nolano AI, the University of Montreal and Mila published Spectra on July 17, 2024, a suite of 54 language models covering nine parameter counts from 99M to 3.9B, each trained on the same 300B tokens of SlimPajama. Every size exists as a half-precision model the authors call FloatLM, as GPTQ-quantised copies at 3, 4, 6 and 8 bits called QuantLMs, and as a TriLM, a model trained from scratch with every linear-layer weight restricted to -1, 0 or 1. The paper’s size table puts TriLM 3.9B at 10.76 billion bits, roughly 1.3 GB, against 63.83 billion bits for the half-precision model of the same parameter count, a factor of 5.9.

That is the comparison the suite is built for, since what fits on a device is bits rather than parameters. Averaged over six commonsense and reasoning benchmarks, the paper’s tables put TriLM 3.9B at 60.7 against 61.4 for FloatLM 3.9B, level with the 4-bit QuantLM 3.9B at 60.7 which needs 20.59 billion bits. Set against the models nearest to it in size, TriLM 3.9B beats the half-precision 830M model, 60.7 against 53.3 at 13.34 billion bits, and the 3-bit QuantLM 2.4B, 60.7 against 54.7 at 10.95 billion bits. On LAMBADA it reaches 61.6 against 61.1 for FloatLM 3.9B, and on TriviaQA 21.3 against 21.5.

During training a TriLM keeps floating-point latent weights and ternarises them on the fly in the forward pass, with one scale value per weight matrix shard and gradients passed through by a straight-through estimator, so only the ternary states and the scales are needed at inference. Embeddings and the output head stay in half precision across the suite. The authors position TriLM against BitNet b1.58, which also trains ternary weights from scratch, and say they normalise once before each linear layer instead of normalising, scaling and quantising activations to 8 bits four to seven times per transformer block; their own replication of the BitNet architecture at 1.1B, trained on 100B tokens, scores below TriLM 1.1B on the same benchmarks.

The paper names the weaknesses it measured. At 1.1B parameters the ternary model does not reach its half-precision counterpart, 51.6 against 54.9 on the six-benchmark average, and the 2.4B one still trails at 57.3 against 59.2, so the match with a half-precision model of the same parameter count holds at 3.9B only. On CrowsPairs, which measures how often a model prefers a stereotyped sentence, TriLM 3.9B lands at 65.4 percent against 64.7 for FloatLM 3.9B and 62.6 for the smaller FloatLM 830M, so the ternary model stereotypes more than the half-precision model closest to it in bits. No model in the suite beats random guessing on TruthfulQA, and while TriLM 3.9B holds up on Penn Treebank and LAMBADA perplexity, it stays behind FloatLM 3.9B on web corpora including Dolma, C4 and RefinedWeb, which the authors read as weaker memorisation of training data.

The weights and more than 500 intermediate checkpoints are on Hugging Face under Apache 2.0. The TriLM checkpoints are published unpacked to FP16, which the model cards say makes them compatible with ordinary half-precision matrix multiplication and leaves the released files at full half-precision size. The authors list smartphones, laptops and automobiles among the resource-constrained devices they see for ternary models, and end by calling for work on the toxicity, stereotyping and web-corpus gaps they measured.