0
$\begingroup$

I'm reading the famous attention model paper Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, & Bengio, 2014) and I'm having trouble what the exact meaning of "align" is in this context.

I've read this Quora answer which does a great job explaining the overall model, but doesn't address what this word means in this context.

I'm asking because it seems to have a different meaning than the conventional definition of putting things in a straight line.

Would anyone be kind enough to explain the meaning to me? Thank you.

$\endgroup$
1
$\begingroup$

Consider the following sentence in english and german (taken from wikipedia):

He has eaten an apple.

Er hat einen Apfel gegessen.

The literal translation of the german sentence is

He has an apple eaten.

Suppose you're training an english -> german translation network. Suppose the network has already generated the first 3 words

Er hat einen

You might intuit that when generating "Apfel", the attention of the model should be focused on the english word "apple". However, "Apfel" is the 4th word in the german sentence but the fifth word in the english sentence. So a naive strategy like always focusing on the $k$th word of the source sentence when generating the $k$th word of the target would not be sufficient.

This ability of the model to figure out that it should pay attention to the 5th word of the english sentence to generate the 4th word of the german one is called alignment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.