Breakthrough Proof Explains Why RLHF Alignment Remains Shallow in Large Language Models

In a groundbreaking arXiv preprint released on March 5, 2026, researcher Robin Young from the University of Cambridge has provided the first rigorous theoretical explanation for why Reinforcement Learning from Human Feedback (RLHF) produces shallow safety alignment in large language models (LLMs). The paper, titled "Why Is RLHF Alignment Shallow? A Gradient Analysis," demonstrates that gradient-based alignment inherently concentrates optimization efforts on early positions in a sequence where harm is determined, leaving later tokens effectively unaligned. This resolves a long-standing puzzle in AI safety research, where empirical observations showed KL divergence between aligned and base models decaying rapidly after initial tokens, making models vulnerable to attacks like prefilling.

Shallow alignment has been a critical concern in AI safety, as it means LLMs appear safe during standard evaluations but fail to maintain safeguards deeper into generation. Young's analysis employs a martingale decomposition of sequence-level harm, defining conditional expected harm \( h_t \) at each position \( t \) and innovations \( \Delta_t \) that capture how each token influences overall harmfulness. The key metric introduced is "harm information" \( I_t = \mathbb{E}[\Delta_t^2] \), quantifying a position's impact on harm variance. Beyond the "harm horizon"—the point where harm is fully determined—gradients vanish entirely, proving that standard RLHF objectives like PPO cannot achieve deep alignment no matter the training quality.

The core theorem states that the alignment gradient at position \( t \) is the covariance between conditional expected harm and the score function: \( \nabla_\theta J(t) = \text{Cov}( \mathbb{E}[H | x_{ k \), this covariance is zero, ensuring no training signal reaches later positions. At equilibrium, per-position KL divergence scales as \( \mathcal{O}(\lambda^2 I_t) \), confirming that alignment depth tracks harm information distribution, which empirical data shows biases toward prefixes.

This discovery has profound implications for AI safety, exposing why current methods leave LLMs susceptible to adversarial prefilling attacks that bypass early refusals. It reframes shallow alignment not as an optimization failure but as the mathematically optimal outcome of sequence-level objectives, urging a shift toward position-aware training. Without intervention, scaling LLMs will amplify these vulnerabilities, as longer sequences exacerbate the harm horizon problem.

Young proposes a novel "deep alignment objective" incorporating recovery penalties, which penalize failure to emit safe recovery tokens (e.g., refusals) at every position, generating uniform gradient signals. Weighted by position and discounted over length, this approach theoretically justifies empirical successes like token-level data augmentation. By providing a pathway to robust, deep safety alignment, the work paves the way for more reliable AI systems amid accelerating capabilities.