I'm currently trying to implement a PyTorch version of the Transformer and had a question.

I've noticed that many implementations apply a mask not just to the decoder but also to the encoder. The official TensorFlow tutorial for the Transformer also states that the Transformer uses something called "MultiHead Attention (with padding masking)."

I'm just confused, why are masks applied to the padding in the encoder sequence?


Just an example why people want to apply masks to encoders.

There're unsupervised language models pre-trained with an unidirectional mask, for example GPT. If we want to leverage this pre-trained language model to build a encoder-decoder based machine translation model, we might want to apply the unidirectional mask in the same fashion it's pre-trained with.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.