📢 Wondering how the neocortex works, how it is related to modern machine learning algorithms, and how this insight can be used to fuel next-gen neuromorphic hardware?
Have a look at this PhD opening in my team: tudelft.nl/over-tu-delft/…
Position open until filled, apply early!

Interested in Computational Neuroscience in 🇨🇭?

Looking for a PhD or masters thesis project? 👀 Check out the impressive network of labs!

#neuroAI #compneuro #AcademicTwitter #neuroTwitter #neuroX #machinelearning

swisscompneuro.org

S4, Mamba, and Hawk/Griffin are great – but do we really understand how they work? We fully characterize the power of gated (selective) SSMs mathematically using powerful tools from Rough Path Theory. All thanks to our math magician Nicola Muça Cirone

arxiv.org/pdf/2402.19047…
🧵

Google presents Griffin

Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN

We’ve all heard of place cells: neurons that form a “cognitive map” representing the structure of the outside world. But what about our own structured behaviours? We took a deep mechanistic dive, out now on bioRxiv : biorxiv.org/content/10.110… 🧵below:

So (1) RNNs were not used anymore for computational reasons, (2) the recurrent process does not have to be 'smart', (3) parallel scan is the stuff.

Curious how modern RNNs/ state-space models (à la MAMBA) enable online learning of long-range dependencies?

I will be presenting our poster in two hours (#719), come check it out!