LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

arXiv:2506.17871v4 Announce Type: replace-cross Abstract: Despite their impressive capabilities, aligned large language models (LLMs) often generate outputs that lack diversity. What drives this consistency in the generation? We investigate this phenomenon through the lens of probability concentration in the model's output distribution. To quantify it, we use the Branching Factor (BF)--the exponentiated length-averaged entropy of the output distribution, interpreted as the effective number of plausible next steps during generation. Our empirical analysis reveals two key findings: (1) BF often decreases as generation progresses, suggesting that LLMs become more predictable; a controlled intervention indicates that this decline is largely a task-independent property of autoregressive self-conditioning, distinct from alignment, and can be locally reversed by unexpected context. (2) Alignment tuning sharpens the output distribution from the outset, reducing BF by a factor of 2--5 overall and up to an order of magnitude (e.g., from 12 to 1.2) at early positions. This reduction helps explain why aligned models are less sensitive to decoding strategies. It also has implications for complex reasoning: aligned Chain-of-Thought (CoT) models (e.g., DeepSeek-distilled models) generate longer reasoning chains that reach later, more deterministic (lower-BF) stages, yielding more stable outputs. We hypothesize that alignment does not fundamentally change model behavior, but instead steers the model toward stylistic tokens (e.g., ``Sure'') that unlock low-entropy trajectories already present in the base model. Nudging experiments support this view: prompting base models with such tokens similarly reduces BF. Together, our findings establish BF as a diagnostic for understanding and controlling LLM outputs, clarifying how alignment reduces variability, CoT stabilizes generation, and base models can be steered away from diversity.