Octopus fine-tunes a 2B model to 93 percent on API function calls

Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on.

The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure.

The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard.

Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes.

The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license.