Edge0 releases MoE expert-offloading framework, demos 35B model on iPhone

Samuel Zeng has released Edge0, an open-source framework for streaming mixture-of-experts inference, under Apache-2.0. In his launch post he reports a 35-billion-parameter model running on an iPhone at 1 to 2.5 GB of peak memory without a cloud connection.

The repository describes three techniques. Expert weights are streamed from storage on demand, so peak memory is bounded by the active set. A trained prerouter predicts expert routing one step ahead so that loads overlap the forward pass, which the project credits with up to 59 percent higher throughput. Recover-LoRA pairs an int4-quantised frozen base with distilled LoRA adapters.

Two tiers are published. edge0-35b has 40 layers and 256 experts, a checkpoint of roughly 23 GB and about 2.9 GiB of peak active memory. edge0-8b has 24 layers and 128 experts, a 4.2 GB checkpoint and about 1.0 GB peak. On a Mac mini M4 Pro with 24 GB, the project measures 14.9 to 17.7 tokens per second of decode for the 35B tier and 23.9 to 25.3 for the 8B tier, with prefill at 113 tokens per second cold and 140 warm for 35B. Quality loss against FP16 baselines is given as 3.9 points on average for the 35B tier and 2.8 points for the 8B tier.

The repository states that the MLX backend runs on macOS with Apple Silicon, on M1 through M4, and requires Python 3.10 or newer.