Apple Researchers Derive Scaling Laws for Pretraining With Scarce Target Data

The data-mixture problem

Apple Machine Learning Research published work in August 2026 on a common pretraining problem: how much scarce, high-value target data should be mixed with abundant generic data when training a language model. This matters for low-resource languages, specialized domains and quality-filtered corpora where the target dataset is useful but too small to scale linearly with model compute.

If the target share is too small, the model underexposes itself to the desired domain. If the same target examples are repeated too aggressively, returns eventually diminish and overfitting becomes a concern. The researchers study where that trade-off changes as model size, dataset size and compute budget vary.

More than 2,000 training runs

The study spans more than 2,000 language-model training runs across multilingual, domain-specific and quality-filtered mixtures. The authors report that repetition is a central driver of target-domain performance, but mixture training tolerates substantially more repetition than single-source training.

Across the studied settings, the paper finds that scarce target corpora can often be reused roughly 15 to 20 times before the optimal mixture shifts, although the best repetition level depends on target-data size, total compute and model scale. This is an empirical finding from the reported experiments rather than a universal rule for every dataset or architecture.

A repetition-aware scaling law

The researchers introduce a scaling law that explicitly models the declining value of repeated target tokens and the regularizing contribution of generic data. Instead of choosing a data mixture through trial and error alone, the formulation is designed to estimate a compute-efficient mixture for a particular target dataset and training budget.

That is potentially useful for teams training multilingual or domain-specialized models because high-quality target data is often the limiting resource. A mixture rule that accounts for repetition can help allocate compute without assuming that each repeated token has the same marginal value as a fresh one.

Practical implications

The work suggests that developers should not automatically discard repeated target data once a scarce corpus has been seen a few times. Under mixed pretraining, generic data may act as a regularizer that lets the model continue benefiting from repeated target examples for longer than expected.

At the same time, the paper does not imply that arbitrary duplication is beneficial. Dataset quality, contamination, architecture, tokenizer behavior and downstream objectives can all change the optimum. The main contribution is a framework for reasoning about these interactions rather than a fixed recipe.

Research status

This is an Apple-affiliated research publication, not a new Apple product or foundation-model release. Its relevance is methodological: it provides an empirically grounded approach to designing training mixtures when valuable target data is constrained.