3
$\begingroup$

Consider this introduction to attention layers with the main description below.

I understand attention layers as learnable soft query retrieval operators that act on a "K-V store" of vectors. A common use case is to learn a "sequence to sequence" task where output words can query the input sequence to soft "align" on the right input sequence word or word context.

What's the intuition behind multi-head attention? How are they used in practice? Do they just compute the same projection multiple times to just get a higher dimensional representation? (I doubt it). Or are the extra heads focused on shifted inputs in any way? What "extra information" or computation do they extract that can be useful for a particular task?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

I'll tell you what I know/have heard about it. There's probably other analysis out there I haven't seen.

I think multi-head attention was introduced in the transformer network paper. Their justification/explanation is:

enter image description here

Another paper on a modified Transformer architecture questions this understanding though:

enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Simon. At a more high-level though, In the original Transformer and related work, are the inputs actually different for each of the attention heads? In the images that I posted I don't see anything indexing over different parts or chunks of the input, and these heads don't seem to be feeding each other either, so what's the point of using multiple heads if they take in the same exact input, even if they just get concatenated in a final vector? Is it to just learn multiple variants of the same circuit that could learn multiple representations or (transformations) of the same input? $\endgroup$
    – Josh
    Commented Jul 2, 2020 at 15:04
  • 1
    $\begingroup$ The inputs are the same for each head, until to the projection—each head gets its own projection to learn. So the arrangement of weights and such is different than for single head attention. That's all I know though. May just be a "well this works better in practice" but nobody knows exactly why $\endgroup$
    – Simon Alford
    Commented Jul 2, 2020 at 15:21
  • $\begingroup$ @Josh The different heads work on different sub-parts of vector-dimensions, 8 heads - 64 dimensions each is what I understand the "standard" setting. That helps parallelization of the execution, one single 512^2 attention head is both memory and processing heavy. $\endgroup$
    – Peter Wirdemo
    Commented Sep 6, 2020 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.