Encoding Layers in the Transformer In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here:

My understanding is that this is an analogue of convolutional neural nets, where we are extracting higher-and-higher level features as we pass through each encoding layer. Initially we're looking for simple things like verbs, nouns, etc. Then we're looking for subject-object relations. Then relations between subjects, and if the phrases are longer, relations between phrases.
Looking that these images however, I'm a bit lost about the actual need for so many attention filters at each layer. The nearly-diagonal attention maps represent dead filters (e.g. features that weren't detected). But, the last layer (6) is most telling: there is a vertical collapse of  nearly all filters. 
Question: Why is this vertical collapse happening in the final encoder layer 6? Is it again some form of non-detected features? Why isn't the collapse diagonal?
 A: Recently there were two papers commenting on the self-attention heads:


*

*Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned from University of Edinburgh

*Are Sixteen Heads Really Better than One? from CMU
Both of them come to the same conclusion that most of the heads are just noise and can be relatively easily pruned out.
The Edinburgh guys also spotted some regularities in what the heads do: they mostly attend to adjacent or more distant neighboring words (after all the Transformer is not explicitly aware of the mutual position of the words and need to learn it), some tend to capture some syntactic relations (verb to adverbial modifier, subject to verb, etc.), some to tend to attend to rare words..
There is also a paper attempting to extract constituency syntax from self-attentions.
A: If you look carefully most of the diagonal attention patterns are actually offset a bit from diagonal, indicating most of the attention is on nearby tokens, which is expected. The authors do say in the paper that it might be possible to limit the attention neighborhood to save on computation. But the layer 4 attention maps do show that being able to attend across the entire sentence can be useful.
As for layer 6, I'm not sure entirely. Is breaking up "secretly" into "secret" and "ly" intended or a mistake? I'm not a NLP expert but I've never heard of such a tokenization step. The tokenizer they use, spacy, doesn't break the word up either. If this was a mistake, then that would explain it.
There are also some attention visualizations in the original transformer paper which you might find interesting. I don't think they "collapse" there unexpectedly. They also show more interesting structure rather than just connections on the diagonal.
