LaMini-LM distils models from 61M parameters up on 2.58M instructions

Researchers at MBZUAI, the University of British Columbia and Monash University published LaMini-LM on April 27, 2023, a family of 17 distilled instruction-following models ranging from 61M to 7B parameters. The paper gives resource use as the motivation, stating that earlier distilled models of 7B to 13B parameters present “challenges for deployment in resource-constrained settings” and that the authors set out to make deployment in such settings easier. The paper names no target device and reports no memory footprint, no quantisation and no generation speed, and the authors state they trained and evaluated on V100 and A100 GPUs.

The models are fine-tuned on the LaMini instruction dataset, 2.58M instruction and response pairs that the team generated with gpt-3.5-turbo from prompts taken from self-instruct, P3, FLAN and Alpaca. The authors describe it as the largest instruction dataset of its kind at the time and 50 times the size of the dataset behind Alpaca. The students are 6 encoder-decoder models built on T5 and Flan-T5 and 11 decoder-only models built on GPT-2, GPT-Neo, Cerebras-GPT, GPT-J and LLaMA.

The authors evaluated the family on 15 NLP benchmarks with EleutherAI’s lm-evaluation-harness and report that LaMini-Flan-T5-248M performs on par with LLaMA-7B on the benchmark average, against a baseline with about 28 times as many parameters. They warn in the repository that their LLaMA figures are not comparable to the ones the LLaMA authors published, because they re-ran the evaluation themselves. A human rating of 114 user-oriented instructions follows the same size trend and leaves a gap to gpt-3.5-turbo, and the paper reports that the models do poorly on coding, maths and tasks that need reasoning.

MBZUAI publishes the weights and the instruction dataset on Hugging Face. The repository puts the code under Apache 2.0 and the models and the data under CC BY-NC 4.0, and states that both are intended for research use only. The paper was later accepted at the EACL 2024 main conference.