Distillation
11 updates on Distillation.
Online-SDFT reports 70.28% routing accuracy and updates a LoRA adapter on the phone
I-Ju Lin and Zhang-Wei Hong measure 70.28% accuracy against 52.78% for the best baseline, and run the LoRA update itself on an Android phone.
Apertus Mini distils an open-data 8B into 0.5B, 1.5B and 4B models
Swiss AI distilled its fully open Apertus 8B into 0.5B, 1.5B and 4B models on 1.7T tokens, with 3-bit to 6-bit MLX builds for Apple devices.
Tempo compresses hour-long video with a 2B vision model so a 4B LLM can answer
Meta AI and KAUST report 52.7 on LVBench for a 6B system in which a 2B vision-language model squeezes long video down to about 3 tokens per frame.
Meta's MobileLLM-Pro runs a 128k context from a 590 MB 4-bit build
Meta Reality Labs released a 1.08B on-device model with a 128k context window, measured at 33.6 tok/s decode on a Galaxy S25 CPU.
Meta trains 140M to 950M reasoning models on 4.2T tokens
MobileLLM-R1 spans 140M to 950M parameters, trained on 4.2T tokens, and Meta scores the 950M model at 74.0 on MATH500.
Apple puts the cost of 2-bit compression at 3.4 MMLU points
Apple measures its on-device model at 67.8 MMLU in 16 bits and 64.4 after compression to 2 bits per weight, and details the distillation pipeline behind it.
Amazon survey puts some small models at 10 to 100 times their parameter count
A survey of about 160 papers finds 1B to 8B models matching much larger ones on single benchmarks, and shows where models below 1B fall apart.
Gemma 2 2B is distilled from a larger model and scores 1126 on Chatbot Arena
Google released a 2.6B-parameter Gemma 2 trained by distilling a larger model, and reported an Elo of 1126 on the LMSYS Chatbot Arena.
LaMini-LM distils models from 61M parameters up on 2.58M instructions
MBZUAI distilled 17 instruction models from 61M to 7B parameters on 2.58M synthetic instructions, and reports the 248M model matching LLaMA-7B.
Hugging Face's DistilBERT is 40% smaller than BERT and 71% faster on an iPhone
The 66M-parameter distilled BERT keeps 97% of its teacher on GLUE and weighs 207 MB inside a question-answering app on an iPhone 7 Plus.
TinyBERT shrinks BERT-base to 14.5M parameters and runs 9.4x faster
Huawei's distilled BERT keeps more than 96.8% of its teacher's GLUE score at 14.5M parameters, 7.5x smaller and 9.4x faster on inference.