S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

Reinforcement learning from human feedback (RLHF) often exhibits unstable training dynamics. One contributing reason is that RLHF relies on a single sequence-level scalar reward that is propagated to token-level updates during optimization, making credit assignment inherently ambiguous. Recent work explores reward refinement to provide denser supervision in RLHF, often motivated by increasingly accurate reward decomposition and the implicit assumption that finer-grained credit assignment improves optimization. We argue that this assumption is suboptimal: overly fine-grained reward refinement can amplify noise and uncertainty in the reward signal, ultimately degrading model performance and training stability. In light of this, we propose a granularity-aware reward refinement principle that emphasizes stability-oriented reward design rather than maximal allocation precision. Under this principle, we identify sentences as a natural intermediate granularity for reward allocation, balancing robustness to noise with the ability to convey semantically meaningful supervision. Guided by this principle, we introduce S2T-RLHF, a sentence-to-token reward decomposition framework that enforces structured, semantically grounded credit assignment while requiring no reward-model retraining or token-level supervision. Experiments across multiple datasets demonstrate that S2T-RLHF yields more stable optimization dynamics and improved robustness and generalization.