Trust Region Policy Distillation
Big goals are hard to achieve all at once; breaking them into small steps is wiser.
Hugging Face · Daily Papers
·Zhengpeng Xie, Li Lyna Zhang
·
·▲ 17 upvotes
Este artigo está em destaque na seleção diária de papers do Hugging Face, curada pela comunidade de pesquisa em IA.
Autores: Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, Mao Yang
- 17 upvotes da comunidade
Resumo
Resumo original (em inglês), extraído do paper:
Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.Onde ler
// relacionados
