Apple PROOF-Gen Turns Failed Agent Traces Into Better Distillation Data

Apple researchers publish PROOF-Gen

Apple Machine Learning Research has published PROOF-Gen: From Optimized Data to Better Distillation, a method for improving the data used to distill tool-calling behavior from large teacher models into smaller deployable models.

The paper was submitted to arXiv on August 24, 2026 and is listed by Apple as an EMNLP paper. It is research, not a newly released Apple model or developer product.

The work focuses on a common post-training pipeline: ask a strong teacher model to solve tool-using tasks, keep trajectories that pass evaluation and discard the failures. The authors argue that this generate-and-filter approach wastes useful information because many failed runs are near-misses rather than completely unusable attempts.

How PROOF-Gen works

PROOF-Gen stands for Per-scenario Reflective Optimization to Overcome Failed Generation.

For each failed task, a reflector analyzes the execution trace and evaluator feedback, then writes corrective guidance intended to steer the teacher toward a successful trajectory on another attempt. Once a successful trajectory is recovered, the temporary guidance is removed before student training.

That last step matters: the student receives a clean demonstration rather than a task-specific hint that would not exist at deployment time.

The technique is aimed at tool-calling agents where training data can be expensive because every new teacher trajectory may require a frontier model plus external tool execution and evaluation.

Reported results

On τ2-bench, the paper says 57% of teacher trials failed, with roughly two-thirds of those failures classified as near-misses. PROOF-Gen reportedly recovered successful trajectories for 93% of failed scenarios.

When the recovered data was added to fine-tuning, the authors report that Qwen3-4B-Instruct-2507 improved from a Pass@1 score of 0.132 to 0.529. They also report a 7.2 percentage-point gain for Gemma 4 E4B-it on BFCL v4 multi-turn.

The Apple research page further states that, in a deployed pipeline, the method improved trajectory quality by 6.3 percentage points in goal completion and transferred to a deployed on-device model, including positive average gains across non-English locales.

These numbers are research-paper results under the authors' evaluation setup. They should not be generalized to every agent stack, teacher model or tool environment without independent testing.

Why it matters for agent training

Agent developers increasingly need smaller models that can call tools reliably without paying frontier-model inference costs for every task. Distillation is one path to that goal, but tool-use trajectories are expensive and failures are often discarded.

PROOF-Gen reframes those failures as training opportunities. If a failed trajectory contains mostly correct actions plus one decisive mistake, reflection-guided retrying can potentially recover a useful demonstration instead of spending the entire sample budget again from scratch.

The idea is especially relevant to continuously refreshed agent systems, where teams regenerate demonstrations as APIs, tools and product behavior change.

The status remains important: Apple has published the research and paper, but the cited page does not announce PROOF-Gen as a public framework, API or product feature. Developers interested in the technique should treat it as a research recipe and validate it against their own tool-calling evaluation suite.