The Ultra-Scale Playbook: Training LLMs on GPU Clusters

**Huggingface** released "The Ultra-Scale Playbook: Training LLMs on GPU Clusters," an interactive blogpost based on **4000 scaling experiments on up to 512 GPUs**, providing detailed insights into modern GPU training strategies. **DeepSeek** introduced the Native Sparse Attention (NSA) model, gaining significant community attention, while **Perplexity AI** launched R1-1776, an uncensored and unbiased version of DeepSeek's R1 model. **Google DeepMind** unveiled PaliGemma 2 Mix, a multi-task vision-language model available in **3B, 10B, and 28B sizes**. **Microsoft** introduced Muse, a generative AI model trained on the game Bleeding Edge, and presented Magma, a foundation model for multimodal AI agents excelling in UI navigation and robotic manipulation. **Baichuan-M1-14B** was announced as a state-of-the-art medical LLM trained on **20T tokens**, and a fully open-source 40B genome modeling model using StripedHyena 2 architecture was also released. *"Making your own gaming experience is coming sooner than you'd think,"* noted in relation to Muse.