D2MoE picks a bit-width per token, 1.39 times the throughput at up to 53 percent less memory
Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput.
A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget.
That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB.
A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones.
