Introducing The Hallucinations Leaderboard! 🚀 An open effort to measure the LLMs’ tendency to generate hallucinations across various tasks, like open-domain QA, instruction following, and summarisation! 🧵1/N

📄Paper:arxiv.org/abs/2404.05904
🤗Leaderboard:huggingface.co/spaces/halluci…

Yi Tay Yu Zhao Yeah! However, other (open-source) pre-training code bases we analysed were doing plain causal masking, where the likelihood of each token was conditioned on all previous tokens in the pre-training chunk. This (sub-optimal, as we found in arxiv.org/abs/2402.13991) choice probably…

Fine-tune your LLM unless you have access to GPT-4!
In our SemEval 2024 NLI4CT solution, we evaluated LLMs with ICL, CoT, and a novel PEFT method. And yet, GPT-4 produces surprisingly good results, ranking joint-first in the leaderboard!
👉 arxiv.org/abs/2404.00484
🧵 1/9

Llama 3 was trained using intra-document causal masking, as suggested by Yu Zhao's paper 'Analysing The Impact of Sequence Composition on Language Model Pre-Training'! 🚀🚀🚀 arxiv.org/abs/2402.13991

Congrats to Aaditya Ura ( looking for PhD ), Pasquale Minervini 🚀 looking for postdocs!, Aryo Pradipta Gema and andreas motzfeldt for this cool initiative!
Open Life Science AI

Leaderboard: huggingface.co/spaces/openlif…

Emile van Krieken Chendi Qian Mathias Niepert Christopher Morris Zhe Zeng kareem ahmed Guy Van den Broeck Daniel Daza Thiviyan Thanapalasingam Peter Bloem (@[email protected]) Taraneh Younesian I also got reports that Adaptive IMLE from Pasquale Minervini 🚀 looking for postdocs!, L. Franceschi, and Mathias Niepert works very well for some graph-related sampling tasks, but I haven't tried it out myself yet
arxiv.org/abs/2209.04862

HRQ-VAE is now available in Pythae!

Hierarchical Residual Quantization learns a recursive clustering of a dense vector space, end-to-end. And it learns more meaningful clusters than doing quantization post-hoc.

📝 Paper: aclanthology.org/2022.acl-long.…
🤖 Code: github.com/clementchadebe…

If you're working on LLM pre-training, you may want to take these findings into account! E.g., intra-document causal masking is a one-line drop-in replacement of 'classic' causal masking; it's fully supported by e.g., Flash Attention (github.com/Dao-AILab/flas…); and provides…

🚨New paper🚨
Excited to share that 'Select and Summarize: Scene Saliency for Movie Script Summarization' has been accepted at the #NAACL2024 Findings. 🎬
w. Frank Keller
arxiv.org/abs/2404.03561…

Excited to share our latest work on improving LLM pre-training! 🚀 The amazing Yu Zhao et al. found that focusing on how pre-training sequences are composed and attended over can significantly improve the generalisation properties of LLMs on a wide array of downstream tasks,…

[CL] Learning to Plan and Generate Text with Citations
arxiv.org/abs/2404.03381
- Long-form generation in information-seeking scenarios has seen increasing demand for verifiable systems that generate responses with supporting evidence like citations. Most approaches rely on…

Thanks for coming over to the other side of town Alessandro Suglia!! 🚀🚀🚀🤗🤗🤗

📢 New paper at NAACL:

🤔 > LLMs still struggle with complex reasoning problems. Can we use reasoning nuggets from mathematical proofs to help with it?

Introducing LeanReasoner, a framework that uses Lean to solve complex natural language logical reasoning problems 🧵

New paper:

Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks
arxiv.org/abs/2403.09832

If you are curious to discover more about Dynamic Memory Compression, I will give a preview during my keynote talk at the MOOMIN workshop eaclmeeting

See you on Thursday, March 21st at 9:30 AM!

moomin-workshop.github.io/program

Next week I’ll be in Malta 🇲🇹 to present our work on Improving Generalization in Semantic Parsing by Increasing Natural Language Variation at #EACL2024 !

1/3

Dynamic Memory Compression (DMC) retrofits LLMs like Llama 2 7/13/70B into models with the same downstream performance but with +370% throughput (tok/s) during generation.

*Conditional computation in NNs: principles and research trends*
by Alessio Devoto Valerio Marsocci Jary Pomponi Pasquale Minervini 🚀 looking for postdocs!

Our latest tutorial on increasing modularity in NNs with conditional computation, covering MoEs, token selection, & early exits.

arxiv.org/abs/2403.07965

CR for TRAM is now live! See you at #ICLR2024 in Vienna (as a spotlight poster)

now feat.
* Vision exps (Better Imagenet→CIFAR/Cars/Flowers transfer)
* +ablations (XL model, weird combos)
* Pictures (see below!)
w/ Naomi Saphra Pradeep Dasigi Hao Peng

openreview.net/forum?id=kxebD…

Can open-source LLMs execute *chains of instructions* in a single query? Not so well, we found.

However, they can learn this ability by:
- augmenting examples from public SFT mixtures with chains of instructions automatically
- performing *sequential instruction tuning* on them.…