Gemma 2B and 7B open the Gemma line, built on Gemini research

Google released Gemma on February 21, 2024, the first two models in the line, at 2B and 7B parameters and in a pretrained and an instruction-tuned checkpoint each. The technical report presents the 7B as a model for deployment on GPU and TPU and the 2B as one for CPU and on-device applications. Google says Gemma was built from the same research and technology used to create Gemini and shares technical and infrastructure components with it.

Both sizes are decoder-only transformers trained on a context length of 8192 tokens. The 2B has 18 layers, a model dimension of 2048 and a single key-value head, since Google’s ablations found multi-query attention works well at small scale, while the 7B has 28 layers and keeps standard multi-head attention across 16 heads. Both inherit Gemini’s 256k-entry vocabulary, which puts 524M of the 2B’s parameters into embeddings and leaves 1.98B elsewhere. Google reports training the 2B on 3T tokens and the 7B on 6T, mostly English web documents, mathematics and code.

Google reports Gemma 7B at 64.3 on MMLU 5-shot against 54.8 for Llama-2 13B, 46.4 on GSM8K against 28.7, and 32.3 on HumanEval against 18.3, for an average of 56.9 across 18 academic benchmarks. Google puts the 2B at 42.3 on MMLU and 45.0 on average, ahead of Llama-2 7B on the mathematics and coding tasks and behind it overall. Google states that Gemma outperforms similarly sized open models on 11 of the 18 text-based tasks, and notes it could not rerun the Llama-2 evaluations itself because of that model’s licensing, so it cites Meta’s published figures.

The weights went up on Kaggle and Hugging Face, with the models also runnable from Colab and Vertex AI, and Google provided toolchains for inference and supervised fine-tuning across JAX, PyTorch and TensorFlow through native Keras 3.0. Hugging Face added Gemma to Transformers 4.38 on announcement day. llama.cpp merged Gemma support the same day, within half an hour of the pull request opening, which is what brought the 2B into quantised local runs on consumer hardware. Google states that the models run across laptop, desktop, IoT, mobile and cloud.

The weights are open but the licence is not a standard open source one. Google publishes them under its own Gemma Terms of Use, which allow use, modification and redistribution provided that downstream recipients get the same terms and the separate Prohibited Use Policy, and which reserve Google’s right to restrict uses it considers non-compliant. Google says the terms permit responsible commercial usage and distribution for all organisations regardless of size. Alongside the models Google shipped a Responsible Generative AI Toolkit with a safety classification method, a tool for debugging model behaviour and written guidance for model builders.