In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here: enter image description here

My understanding is that this is an analogue of convolutional neural nets, where we are extracting higher-and-higher level features as we pass through each encoding layer. Initially we're looking for simple things like verbs, nouns, etc. Then we're looking for subject-object relations. Then relations between subjects, and if the phrases are longer, relations between phrases.

Looking that these images however, I'm a bit lost about the actual need for so many attention filters at each layer. The nearly-diagonal attention maps represent dead filters (e.g. features that weren't detected). But, the last layer (6) is most telling: there is a vertical collapse of nearly all filters.

Question: Why is this vertical collapse happening in the final encoder layer 6? Is it again some form of non-detected features? Why isn't the collapse diagonal?


2 Answers 2


Recently there were two papers commenting on the self-attention heads:

Both of them come to the same conclusion that most of the heads are just noise and can be relatively easily pruned out.

The Edinburgh guys also spotted some regularities in what the heads do: they mostly attend to adjacent or more distant neighboring words (after all the Transformer is not explicitly aware of the mutual position of the words and need to learn it), some tend to capture some syntactic relations (verb to adverbial modifier, subject to verb, etc.), some to tend to attend to rare words..

There is also a paper attempting to extract constituency syntax from self-attentions.

  • 1
    $\begingroup$ Thanks! The linked papers are super helpful. $\endgroup$
    – Alex R.
    Commented Jun 6, 2019 at 17:16

If you look carefully most of the diagonal attention patterns are actually offset a bit from diagonal, indicating most of the attention is on nearby tokens, which is expected. The authors do say in the paper that it might be possible to limit the attention neighborhood to save on computation. But the layer 4 attention maps do show that being able to attend across the entire sentence can be useful.

As for layer 6, I'm not sure entirely. Is breaking up "secretly" into "secret" and "ly" intended or a mistake? I'm not a NLP expert but I've never heard of such a tokenization step. The tokenizer they use, spacy, doesn't break the word up either. If this was a mistake, then that would explain it.

There are also some attention visualizations in the original transformer paper which you might find interesting. I don't think they "collapse" there unexpectedly. They also show more interesting structure rather than just connections on the diagonal.

  • $\begingroup$ It's intended to treat common word endings, such as "-ly", "-es" and so on, as tokens of their own. It's a technique to decrease the tooken vocabulary length. As for layer 6, that's the "result" for that specific sentence and such visual collapse is a sign that the model "has made up it's mind" about what token it should output next. Seems like it came to the conclusion that "ly" is the most important token in that decision. $\endgroup$ Commented Sep 3, 2020 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.