BlueLM-V-3B runs a multimodal model on a Dimensity 9300 NPU in 2.2 GB
Researchers at CUHK MMLab and vivo AI Lab published BlueLM-V-3B on November 16, 2024, a vision-language model designed together with the code that runs it on a phone. The authors deployed it on the NPU of a MediaTek Dimensity 9300 in a vivo X100 and report a peak memory requirement of 2.2 GB and a generation speed of 24.4 tok/s with the language model weights quantised to 4-bit.
The model pairs a 2.7B in-house BlueLM language model with a 400M SigLIP vision encoder that takes 384 by 384 image tiles. The authors report an average score of 66.1 on OpenCompass, a suite of eight multimodal benchmarks, which they state is the highest among models of 4B parameters or fewer, ahead of the 8B models MiniCPM-V-2.6 and InternVL2-8B.
Most of the speed comes from how the model decides what resolution to feed the encoder. Dynamic resolution schemes pick a grid of tiles to fit an image, and the authors show that existing choices enlarge pictures a lot, with LLaVA-NeXT resizing and padding a 394 by 390 image to 768 by 768, four times the area, and InternVL 1.5 stretching a 380 by 76 image to 1920 by 384, twenty-five times the area. More pixels mean more tiles and more image tokens, so BlueLM-V-3B uses what the authors call relaxed aspect ratio matching, which tolerates a slightly smaller effective resolution when it wastes less of the canvas. Both examples then land on a single 1:1 tile, and across the 665k-sample LLaVA training set the method picks a smaller grid than InternVL 1.5 in 523k cases.
The rest is shaped around what an NPU is bad at. The authors encode image tiles in batches of four at a time, which they measured as the fastest setting, and run the SigLIP convolution layer on the CPU in parallel with the transformer blocks on the NPU to hide 200 ms of latency. A downsampler merges every 2 by 2 patch into one token, cutting 729 tokens per tile to 196, and the prefill runs in chunks of 128 tokens rather than all at once. Precision is mixed, with INT4 for the language model weights, INT8 for the vision encoder and projector weights and for the KV cache, and INT16 and FP16 activations, which the authors say keeps accuracy stable because activations are more sensitive to quantisation than weights.
Against the 8B MiniCPM-V 2.5 running on the CPU of the same chip through llama.cpp, the authors report 2.53 s to load and encode a 768 by 1536 image and 2.7 s of prefill, versus 4.0 s and 13.9 s, at 24.4 tok/s against 4.9 tok/s. The paper describes no public release of weights or code, and the authors say future work will target a broader range of mobile devices.