P/D-Device prefills in the cloud and decodes on the phone, cutting TTFT 60%

Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer.

The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device.

The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default.

What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on.

The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud.