Adaptively Sparse Transformers
@emnlp2019 +Gonçalo Correia, André Martins

α-entmax attention
α=1: softmax, α=2: sparsemax, continuous in between.
twist: we learn α for each head, w gradients! Some heads become dense, some sparse.

arxiv.org/abs/1909.00015
github.com/deep-spin/entm…

A nice write-up of the challenges of lemmatization by DeepSPINner Erick! Multilingual examples reveal different complexities hard to imagine if focusing on English.

CfP for the workshop on structured prediction for NLP #SPNLP workshop at #naacl19 is out: submit a research/position/overview paper till March 6! structuredprediction.github.io/SPNLP19
NAACL

DeepSPIN talks at #emnlp2018 !

- Thu, 11:00AM, talk @ Blackbox NLP
Interpretable Structure Induction via Sparse Attention.
Peters/Niculae/Martins.

- Fri, 3:36PM, main conf talk @ ML(3B)
Towards Dynamic Computation Graphs via Sparse Latent Structure.
Niculae/Martins/Cardie.

Towards Dynamic Computation Graphs via Sparse Latent Structure: #emnlp2018 + André Claire Cardie

- marginalize over structured latent vars w/ SparseMAP
- CG a function of discrete structure
- eg latent dependency TreeLSTM

pdf arxiv.org/abs/1809.00653
code github.com/vene/sparsemap…

'Structure Back in Play, Translation Wants More Context'

DeepSPINner André Martins writes on the Unbabel R&D blog his notes from this year's #icml2018 and #ACL2018 :

medium.com/unbabel/icml-a…