Papers arxiv:2606.13657 Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation Published on Jun 11 Submitted by Yu on Jun 15 Authors: Guo Yu , , , , , Abstract On-policy distillation exhibits sparse parameter updates that are distributed across layers and favor FF… …
… Step 1 constructs sampler-referenced policy updates to improve candidate quality, while Step 2 performs inference-gap-aware acceptance to filter updates that do not yield consistent improvements after synchronization. …
… The resulting algorithm, Winner Advantage Policy Optimization WAPO , is almost surprisingly simple: perform clipped policy updates only on positive-advantage completions. …
… Isolated-Policy tends to break on roles that are duplicated in the workflow the 3 voters, the 3 workers , since the copies stack up same-direction updates and over-train that role into a late-stage collapse. …
… The following papers were recommended by the Semantic Scholar API Blockwise Policy-Drift Gating for On-Policy Distillation 2026 Reward-Gated On-Policy Distillation 2026 Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models 2026 Prefix-Guided On-Policy … …
Papers arxiv:2605.12070 Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction Published on May 12 Submitted by guanzhong on May 13 jingdong Authors: , , , , , , , Abstract Asynchronous reinforcement learning in large language models faces chal… …
… The How dimension encompasses off-policy, on-policy, and beyond-gradient optimization mechanics. …
… It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE Self-supervised reward model improvement via Value-Anchored On-policy feedback , a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. …
… Inspired by Dynamic sAmpling Policy Optimization DAPO , which improves RL training efficiency by filtering out ineffective rollouts with near-zero advantages, we propose Group-Dynamic reward-Decoupled Policy Optimization GD^2PO . …
… BCQ makes the student pick between a correct teacher trace and a plausible wrong student trace, while NCQ aggregates the student’s failures into a single prompt plus the teacher’s traces, surfacing shared failure modes rather than noisy per-step signals. the prompt replay buffer then re-exposes har… …