1
$\begingroup$

In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below?

  1. independent random weight initialization for each head and
  2. attending only to 512/8 dimension per head (where 8 is number of heads)

If incorrect, please help correct what is incorrect?

References

Attention Is All You Need

3.2.2 Multi-Head Attention

Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. enter image description here

Transformers Explained Visually (Part 3): Multi-head Attention, deep dive

enter image description here

What is different in each head of a multi-head attention mechanism?

enter image description here

$\endgroup$
1
  • $\begingroup$ ensemble usually is associated with "taking N high variance predictors, and increasing the bias taking as prediction some combination of the N predictors output", for example the majority or the mean.... here there is no notion of "reducing the variance", actually it's probably the other way around... increasing the heads is used to increase the variance, in order to capture multiple connections (IMO) $\endgroup$
    – Alberto
    Jul 10, 2022 at 10:34

1 Answer 1

1
$\begingroup$

I would not say so. The idea behind ensembles is that the errors ML models make are in part systematic and in part random. When we train different classifiers for ensembling, we assume that the random part of the error will zero out and will get a better result. All the models in the ensemble do the same thing, but with a different random error.

The intuition behind multi-headed attention is quite the opposite. It is there to allow each head to focus on a different type of information in the attention input representation. In NLP, there is a lot of literature on what the attention heads do: they can look at syntactically related words, some heads can look at surrounding words, and some heads can emulate coreference chains (see e.g., Chapter 5 of this book). When trained correctly, each head should do a different thing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.