Really enjoyed showcasing research that we've been doing the past few years at Sheffield. The main premise of the lecture was that LLMs are idiot savants with remarkable abilities, performing well under extreme or non-intuitive settings (e.g. pre-training, info loss etc.)

Here's the diagram. This aggregate view provides some insights that are otherwise obscure to student researchers who are just getting started: nearly all of our manuscripts eventually get accepted somewhere, but the number of resubmissions necessary varies widely.

Reminder that the LxMLS application deadline is 3 days away! lxmls.it.pt/2024

This year with Kyunghyun Cho Albert Gu Mohit Bansal Noah A. Smith Sara Hooker Danqi Chen Desmond Elliott and other amazing speakers.

I'm excited about this project on compositional understanding across multimodal sequences taken from egocentric kitchen activities

📢 Only 10 days left to apply to the Lisbon Machine Learning School in July. Check out our fantastic lineup of speakers 👇

🔗 for more information: lxmls.it.pt/2024/

🔗 to apply: tinyurl.com/288raakv

#LxMLS #MachineLearning #NLP

🤏 Why do small Language Models underperform?

We prove empirically and theoretically that the LM head on top of language models can limit performance through the softmax bottleneck phenomenon, especially when the hidden dimension <1000.

📄Paper: arxiv.org/pdf/2404.07647…
(1/10)

Great work on how to quickly create strong sentence encoders!

👶 BabyLM Challenge is back!
Can you improve pretraining with a small data budget?

BabyLMs for better LLMs
& for understanding how humans learn from 100M words

New:
How vision affects learning
Bring your own data
Paper track

babylm.github.io
🧵

Ever asked yourself what's the best explainability method for ViT Transformer at the moment?

🚨 We present you LeGrad, a Layerwise Explainability GRADient method for large ViT transformer architectures. 🚨

So, grab your ☕️ and 🥐, release your inner 🇫🇷, and join us on our…

I am looking for an ultra-emergency reviewer for ARR to review a paper on speech multimodality. Please reach out if you can help #NLProc

Sadly I won’t make it to #EACL2024 in Malta this year but Wenyan Li will be there to talk about Data Curation for Image Captioning on Monday at 1400 in Marie Louise 2

Reach out if you want to learn more about the opportunity to work with Isabelle Augenstein and myself on multimodal fact checking.

I am thrilled to share that our paper 'Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations' was accepted to LREC-COLING.
📜 arxiv.org/pdf/2402.19133…
❤️ with Oliver Eberle, Tiago Ribeiro, Anders Søgaard and Nora Hollenstein ML Group, TU Berlin CoAStAL NLP

Do you know of any high-quality language-specific pretraining datasets? I’d love to hear about them and why you think they are of high quality.