FineWeb: 15T Tokens, 12 years of CommonCrawl (deduped and filtered, you're welcome)

**2024** has seen a significant increase in dataset sizes for training large language models, with **Redpajama 2** offering up to **30T tokens**, **DBRX** at **12T tokens**, **Reka Core/Flash/Edge** with **5T tokens**, and **Llama 3** trained on **15T tokens**. **Huggingface** released an open dataset containing **15T tokens** from **12 years** of filtered CommonCrawl data, enabling training of models like **Llama 3** if compute resources are available. On Reddit, **WizardLM-2-8x22b** outperformed other open LLMs including **Llama-3-70b-instruct** in reasoning and math benchmarks. **Claude Opus** demonstrated strong zero-shot code error spotting, surpassing **Llama 3**. Benchmarks revealed limitations in the **LMSYS chatbot leaderboard** due to instruction-tuned models gaming the system, and a new RAG benchmark showed **Llama 3 70B** underperforming compared to **GPT-4**, while **Mistral 8x7B** remained strong. Efficient quantized versions of **Llama 3** models are available on **Huggingface**, with users reporting token generation limits around **9600 tokens** on a 3090 GPU. Safety concerns include a UK sex offender banned from AI tool usage and **GPT-4** demonstrating an **87% success rate** exploiting real vulnerabilities, raising security concerns.