Benchmarks
52 updates on Benchmarks.
Meta trains 140M to 950M reasoning models on 4.2T tokens
MobileLLM-R1 spans 140M to 950M parameters, trained on 4.2T tokens, and Meta scores the 950M model at 74.0 on MATH500.
P/D-Device prefills in the cloud and decodes on the phone, cutting TTFT 60%
Huawei researchers split prefill and decoding between cloud and phone, reporting time to first token down at least 60% and cloud throughput up to 15x.
Benchmark of 68 small language models finds architecture outweighs size on device
A study of 68 models from 100M to 5B puts Phi-3 near 70 percent accuracy and finds first-token time and memory tracking architecture, not parameter count.
Apple puts the cost of 2-bit compression at 3.4 MMLU points
Apple measures its on-device model at 67.8 MMLU in 16 bits and 64.4 after compression to 2 bits per weight, and details the distillation pipeline behind it.
Liquid AI releases LFM2, three CPU-first models from 350M to 1.2B
Liquid AI released open-weight models of 350M, 700M and 1.2B parameters and reports 2x faster decode and prefill than Qwen3 on CPU.
Hugging Face's SmolLM3 is a 3B model with 128k context and two reasoning modes
Hugging Face released a 3B model with a 128k context window, six languages and a switchable reasoning mode, along with the full training recipe.
Alibaba gives Qwen3's 0.6B and 1.7B models a reasoning switch
Qwen3-0.6B and Qwen3-1.7B carry the family's switch between a reasoning mode and a fast mode, with 32K context and Apache 2.0 weights.
Google's ML Drift runs Llama 3.1 8B on a phone GPU at 12.7 tokens per second
The paper measures 37.1 decode tokens per second for Gemma2 2B and 12.7 for Llama 3.1 8B on the Adreno 750 GPU of a Samsung S24.
Gemma 3 adds a 1B size that fits in 0.5 GB as an int4 checkpoint
Google released Gemma 3 at 1B, 4B, 12B and 27B with quantisation-aware int4 checkpoints of 0.5 GB and 2.6 GB for the two smallest sizes.
Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini
Helsinki and EURECOM researchers measured 11 models on a Raspberry Pi 5 and a Jetson Orin Nano, where memory and battery gave out before speed did.
Meta's ParetoQ compares five bit widths and puts the accuracy cliff at 1 bit
Meta trained the same models at 1, 1.58, 2, 3 and 4 bits under one recipe and reports 1.58-bit to 3-bit beating 4-bit at matched memory.
Amazon survey puts some small models at 10 to 100 times their parameter count
A survey of about 160 papers finds 1B to 8B models matching much larger ones on single benchmarks, and shows where models below 1B fall apart.