Llama-Mobile Compresses an 11B Vision Model to 2.7-Bit Weights for Arm CPUs

Llama-Mobile targets large multimodal models on phone-class CPUs

Arm highlighted Llama-Mobile on August 26, 2026, a Graphcore Research collaboration with Arm that explores how to run an 11B-parameter vision-language model on resource-constrained mobile CPUs.

The work compresses Llama 3.2 Vision 11B using a new S3D8 weight format averaging 2.7 bits per parameter and quantization-aware training. Arm reports that the model's weight footprint falls from 21.3 GB in bfloat16 to 3.7 GB, while activations are quantized to INT8.

The underlying paper, submitted to arXiv on August 21, describes the method as a general framework for efficient VLM inference on constrained hardware.

S3D8 stores three weights in one byte

S3D8 is a storage representation designed around Arm CPU vector instructions rather than a new native 2.7-bit arithmetic format.

Three weights share one byte: the format uses a centroid index plus sign information, while learned centroid values are stored in an INT8 table. During inference, Arm Advanced SIMD instructions decode those compressed values into INT8 so existing integer matrix-multiplication instructions can process them.

This separation between compact storage and conventional INT8 computation is important. It reduces memory traffic without requiring a mobile processor to implement dedicated 2.7-bit arithmetic hardware.

Pixel 8a tests show the gain is workload-dependent

Arm tested selected matrix-multiplication workloads on five CPU cores of a Google Pixel 8a. For token generation, S3D8 reached 33.8 GMAC/s, compared with 26.5 GMAC/s for INT8 and 13.6 GMAC/s for bfloat16 in that test.

For text prefill and vision encoding, performance stayed much closer to INT8 because those phases are more compute-bound and S3D8 must first be decoded.

Using a custom C++ runtime, Arm reports a median 3.8 generated tokens per second for the quantized model on the Pixel 8a, excluding model loading and prefill.

These are specific research measurements, not universal phone-performance guarantees. Throughput will vary with CPU generation, core count, memory bandwidth, thermal limits, runtime implementation, prompt length and model configuration.

Quantization-aware distillation recovers much of the lost accuracy

Very aggressive low-bit quantization can sharply reduce model quality. Llama-Mobile uses quantization-aware distillation so the compressed student learns to match the original bfloat16 teacher's next-token distribution while experiencing quantization during training.

A notable part of the method is that it does not require access to the model's original training dataset. The teacher generates responses for ImageNet-sampled images, creating synthetic image-text data used for the distillation stage.

Arm reports an average score of 66.1% across VQAv2, ChartQA, DocVQA and AI2D for the S3D8 model with quantization-aware training, compared with 74.4% for the original bfloat16 model. At the same approximate bit width, the same S3D8 representation with GPTQ scored far lower, highlighting the value of the training-aware approach in this experiment.

Research result, not a new Meta model release

Llama-Mobile does not change the underlying Llama 3.2 Vision model and is not a new foundation-model generation from Meta.

It is an inference and quantization research result focused on the numerical representation, training process and CPU execution path needed to make a large VLM more practical on mobile hardware.

That distinction matters because the headline 2.7-bit figure describes model-weight storage. The runtime still decodes those weights into INT8 operations, and the research reports a measurable accuracy tradeoff relative to the original bfloat16 model.

Why this matters for on-device multimodal AI

Mobile AI increasingly needs to combine cameras, local context and private user data without sending every interaction to a cloud model. But memory footprint and bandwidth remain major constraints for large multimodal systems.

Llama-Mobile shows one path: co-design the quantization format around the processor's existing vector instructions and use training-aware methods to recover quality at very low bit widths.

The work currently demonstrates one model and one main device class. Arm explicitly identifies broader model families, processor generations and multimodal workloads as future research directions, so the result should be treated as a promising engineering method rather than a universal recipe.