Apple team finds H100 last on tokens per dollar for models up to 2B

Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB.

The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64.

FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not.

Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed.

These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence.