Vijay Janapa Reddi argues edge generative AI needs models under 1B parameters
Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud.
Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves.
Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org.