Meta GEM: How LLM-Scale Training Is Being Reworked for Ads Recommendation AI
Meta says its GEM ads recommendation foundation model now trains across thousands of GPUs and has doubled end-to-end training efficiency through custom kernels, low-precision training, topology-aware parallelism and memory optimization.
Meta's engineering team published a detailed look at how it trains the Generative Ads Recommendation Model (GEM), the foundation model used in its advertising recommendation systems. Unlike a conventional language model, GEM combines very large sparse embedding tables with dense model components and recommendation-specific data patterns, which creates different compute, communication and memory bottlenecks.
Meta reports that GEM training now runs across several thousand latest-generation GPUs. The company says it raised end-to-end Model FLOPs Utilization to roughly 20-25% while scaling total training FLOPs fourfold over twelve months. Those numbers are Meta's own production measurements, not an independent benchmark, but they illustrate the engineering challenge of making recommendation foundation models scale efficiently rather than simply adding more accelerators.
The work spans several layers of the training stack. Meta describes recommendation-specific attention kernels, mixed ultra-low-precision training, topology-aware five-dimensional parallelism, communication techniques designed to reduce GPU compute contention, activation checkpointing, activation quantization and sequence-length-aware load balancing. The goal is to improve both single-GPU utilization and the fraction of that performance retained when training is distributed across large clusters.
A notable lesson is that techniques optimized for dense LLM workloads do not automatically transfer to recommendation systems. GEM's sparse parameters, jagged sequences and mixed interaction patterns required purpose-built kernels and different communication strategies. Meta also says future optimization cycles may increasingly use AI agents to automate parts of systems tuning.
For AI engineers, the broader takeaway is that frontier-scale AI infrastructure is becoming more workload-specific. Model architecture, numerical precision, kernels, networking, parallelism and memory management increasingly have to be designed together. The performance figures should be read as vendor-reported engineering results, while the underlying systems-design ideas are useful for teams working on large recommendation, ranking and multimodal training workloads.
This article is built from the source material below. Open the originals for full context and the latest updates.
