Hugging Face's DistilBERT is 40% smaller than BERT and 71% faster on an iPhone

Hugging Face published DistilBERT on October 2, 2019, a compressed version of BERT trained by knowledge distillation, where a small student model learns to reproduce the output of a larger teacher. The student carries 66M parameters against BERT-base’s 110M, and Victor Sanh and his co-authors report it retaining 97% of BERT’s score on the GLUE language understanding benchmark while running 60% faster.

The authors tested the model on a phone by building a question-answering app for an iPhone 7 Plus and comparing it against their own BERT-base question-answering model. Excluding the tokenisation step, they measured DistilBERT at 71% faster, and the whole model in that app weighs 207 MB, which they note could shrink further with quantisation. On a server CPU at batch size 1, the same pair takes 410 seconds against 668 for a full pass over the STS-B development set.

Three losses train the student at once. A distillation loss pushes it towards the teacher’s full output distribution rather than only the correct token, a masked language modelling loss keeps BERT’s own pre-training objective in place, and a cosine embedding loss aligns the directions of student and teacher hidden state vectors. The student halves BERT’s layer count, drops the token-type embeddings and the pooler, and starts from every second layer of the teacher, an initialisation the ablation study values at 3.69 points of GLUE macro score. Training ran on eight 16 GB V100 GPUs for about 90 hours on English Wikipedia and the Toronto Book Corpus.

Hugging Face released the trained weights together with the training code in its transformers library, and published the iOS demo separately as Core ML code. The paper was accepted at the fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing, co-located with NeurIPS 2019.