TensorOpera releases Fox-1, a 1.6B model trained on 3 trillion tokens

TensorOpera released Fox-1 on June 13, 2024, a decoder-only language model with 1.6B parameters trained on 3 trillion tokens of text and code. The company says the run followed a three-stage data curriculum at 8K sequence length, without saying what each of the three stages contained.

Fox-1 uses grouped query attention, where several attention heads share one set of key and value vectors to cut memory traffic, with 16 attention heads over 4 key-value heads. TensorOpera describes the model as deeper than others of its size, 78% deeper than Gemma-2B, 33% deeper than Qwen1.5-1.8B and StableLM-2-1.6B, and 15% deeper than Apple’s OpenELM-1.1B.

TensorOpera reports an average of 47.13% across ARC Challenge, HellaSwag, MMLU, TruthfulQA, Winogrande and GSM8k, which it ran in the Open LLM Leaderboard setup on one machine with 8 H100 GPUs. It puts Qwen1.5-1.8B at 46.81%, Gemma-2B at 46.36%, StableLM-2-1.6B at 45.92% and OpenELM-1.1B at 38.28%. Fox-1 leads that group on GSM8k with 36.39% against 34.04% for Qwen1.5-1.8B and 17.06% for Gemma-2B, and trails Qwen1.5-1.8B on MMLU with 43.05% against 47.15%.

The device claim in the announcement is a target rather than a measurement. TensorOpera puts the model’s FP16 memory use at 3,703 MiB, against 5,379 MiB for Gemma-2B, 4,739 MiB for Qwen1.5-1.8B and 3,852 MiB for StableLM-2-1.6B, and says that makes Fox-1 comparable for on-device deployment. The stated plan is that developers train on the company’s cloud platform and then deploy, personalise and monitor models on smartphones and AI-enabled PCs through its FedML platform. No figure for speed or latency on a phone appears in the post, and the throughput number it does give, over 200 output tokens per second per user, was measured with vLLM on a single H100.

TensorOpera published the base model under the Apache 2.0 license on Hugging Face and on its own platform, and said an instruction-tuned version would follow.