Is Transformer multi-headed attention a form of ensemble?

Question

In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below?

1. independent random weight initialization for each head and
2. attending only to 512/8 dimension per head (where 8 is number of heads)

If incorrect, please help correct what is incorrect?

References

Attention Is All You Need

3.2.2 Multi-Head Attention

Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

Transformers Explained Visually (Part 3): Multi-head Attention, deep dive

What is different in each head of a multi-head attention mechanism?

Answer 1

I would not say so. The idea behind ensembles is that the errors ML models make are in part systematic and in part random. When we train different classifiers for ensembling, we assume that the random part of the error will zero out and will get a better result. All the models in the ensemble do the same thing, but with a different random error.

The intuition behind multi-headed attention is quite the opposite. It is there to allow each head to focus on a different type of information in the attention input representation. In NLP, there is a lot of literature on what the attention heads do: they can look at syntactically related words, some heads can look at surrounding words, and some heads can emulate coreference chains (see e.g., Chapter 5 of this book). When trained correctly, each head should do a different thing.