Life after DPO (RewardBench)

**xAI raised $6 billion at a $24 billion valuation**, positioning it among the most highly valued AI startups, with expectations to fund **GPT-5 and GPT-6 class models**. The **RewardBench** tool, developed by Nathan Lambert, evaluates reward models (RMs) for language models, showing Cohere's RMs outperforming open-source alternatives. The discussion highlights the evolution of language models from Claude Shannon's 1948 model to GPT-3 and beyond, emphasizing the role of **RLHF (Reinforcement Learning from Human Feedback)** and the newer **DPO (Direct Preference Optimization)** method. Notably, some **Llama 3 8B reward model-focused models** are currently outperforming GPT-4, Cohere, Gemini, and Claude on the RewardBench leaderboard, raising questions about reward hacking. Future alignment research directions include improving preference datasets, DPO techniques, and personalization in language models. The report also compares xAI's valuation with OpenAI, Mistral AI, and Anthropic, noting speculation about xAI's spending on Nvidia hardware.