torchchat runs Llama 3 8B on a Galaxy S23 and iPhone at more than 8 tok/s

PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch.

The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark.

Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster.

Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb.

The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference.