Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++

Georgi Gerganov pushed the first commits of llama.cpp on March 10, 2023, according to the repository’s commit history. The project is an inference engine for Meta’s LLaMA models written in plain C and C++, and the README states its goal as language model and vision-language model inference with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud.

The core implementation carries no external dependencies. It supports integer quantisation at 1.5, 2, 3, 4, 5, 6 and 8 bits, which is what brings models within reach of consumer memory budgets. Apple silicon is a first-class target through ARM NEON, the Accelerate framework and Metal.

The project has since added backends for x86 with AVX, AVX2, AVX512 and AMX, RISC-V, NVIDIA CUDA, AMD HIP, Moore Threads MUSA, Vulkan, SYCL, OpenCL and WebGPU. It is published under the MIT license.