Agentic reinforcement learning · 2026
Persistent consistency for self-distillation
PCSD learns where a privileged teacher is reliably helpful—not from a single token, but from support that persists across a local neighborhood.
adaptive windowspersistent evidencecontinuous weights
Dense guidance is trusted only when teacher support persists.
The challenge
One outcome.
Hundreds of decisions.
Long-horizon agents often receive a single reward at the end of a multi-turn trajectory. A privileged teacher can add dense token-level guidance, but that teacher is not equally reliable everywhere.
PCSD asks a better question: does the teacher’s support persist?
A single spike can mislead
Pointwise discrepancies are precise, but sensitive to transient noise.
A local pattern earns trust
Nearby, consistent support produces smoother and more credible guidance.
How PCSD works
Local evidence,
carefully weighted.
Three complementary mechanisms turn teacher–student log-probability gaps into continuous token-level distillation weights.
Stable regions keep a short window for precision. Noisy regions borrow a longer window for robustness.
Evidence nearer the current token matters more, preserving locality while suppressing isolated fluctuations.
Declining teacher support is attenuated; a sigmoid gate maps credible support into smooth token weights.
Joint objective
ℒtotal = ℒGRPO + λ ℒPCSD
Sparse trajectory feedback + dense, credibility-aware supervision.
Interactive intuition
See how a wider local window turns scattered signals into persistent evidence.
This lightweight illustration is for intuition; the paper uses adaptive aggregation, exponential decay and trend modulation together.
Local window 4 tokensIllustrative credibility 78%
3144386772797584
preciserobust
Main results
Stronger agents.
No inference-time skills.
PCSD achieves the best ALFWorld overall success rate among all evaluated baselines on both backbones.
+15.6 points over GRPO
+13.3 points over GRPO
+15.8 points over GRPO
Best normalized score
ALFWorld overall success rate
Consistent gains across model scales
PCSD baselines
Qwen2.5-3B-Instruct
Beyond the headline
Why persistence
matters.
Ablations and robustness tests show that every component contributes to reliable weighting—not just aggregate benchmark performance.
Every component counts
Removing adaptive windows, decay, or trend modulation lowers overall performance.
Full PCSD 90.6
Fixed N=4 88.3
Without decay 85.1
Without trend 83.6
Robust under stronger noise
PCSD better preserves token-weight rankings as isolated-gap perturbations increase.
0.934
0.887
Generalizes to unseen worlds
The trained policy needs no privileged skill retrieval during evaluation.
86.7%ALFWorld unseen overall
Citation
Build on persistent consistency.
If PCSD supports your research, please cite the paper using the BibTeX entry provided here.
DOI: 10.48550/arXiv.2608.01837@misc{lv2026pcsdpersistentconsistencyselfdistillation,
title = {PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning},
author = {Chunji Lv and Yangguang Wei and Junlin Liu and Yang Gao and Ming Liu and Xinming Wang and Jinyang Wu and Guoren Wang and Changsheng Li},
year = {2026},
eprint = {2608.01837},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2608.01837}
}