The full training loop metrics are now available on W&B 👇

Stage 1 pretraining: wandb.ai/ai2-llm/OLMo-7…
Stage 2 annealing: wandb.ai/ai2-llm/OLMo-7…

llama3-70b-instruct passes my “am I an entity” test (wrong pronouns aside 🫠)

tired: 'As an AI language model...'
wired: 'Built with Meta Llama 3'

Big announcement: pleias releases a massive open corpus of 2 million Youtube videos in Creative Commons (CC-By) on Hugging Face. Youtube-Commons features 30 billion words of audio transcriptions in multiple languages, and soon other modalities huggingface.co/datasets/PleIA…

de-de-de-dup 😈

notable stuff:
🦉ton of perf boost from mixing instruct data at end (e.g., flan)
🐋anneal learning rate (Fig 9b in arxiv.org/abs/2403.08763)
🐞changing data mix boosts MMLU at some cost to other evals

🍇huggingface.co/allenai/dolma
🧀huggingface.co/allenai/OLMo-1…

Amazing work as always by Luca Soldaini 🎀 and and the folks at AI2. Major improvements on MMLU by tweaking the recipe on data cleaning and mixing. Read the blog and learn from the best. Open science is so important!

Our new position paper showing how social choice theory can give us tools to answer many vexing questions around the preference collection and learning in rlhf!

r/LocalLLaMA is pure poetry

13b models are dope//sorry maccas is ready