# Why do we use masking for padding in the Transformer's encoder?

I'm currently trying to implement a PyTorch version of the Transformer and had a question.

I've noticed that many implementations apply a mask not just to the decoder but also to the encoder. The official TensorFlow tutorial for the Transformer also states that the Transformer uses something called "MultiHead Attention (with padding masking)."

I'm just confused, why are masks applied to the padding in the encoder sequence?

I hadn't realized this question was unanswered. If I were to attempt to answer my own question, we apply masks to the source data because after the data passes through the Encoder sublayer, there are values for the padding sequences. We don't need nor want the model to attend to these padding sequences, and so we mask them out.

It's slightly different from masking in the decoder in the sense that masking in the decoder takes an additional step of having a "no peaking" mechanism so that our model can't look at future tokens.

The mask is simply to ensure that the encoder doesn't pay any attention to padding tokens. Here is the formula for the masked scaled dot product attention: $$\mathrm{Attention}(Q, K, V, M) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}M\right)V$$ Softmax outputs a probability distribution. By setting the mask vector $$M$$ to a value close to negative infinity where we have padding tokens and 1 otherwise, we ensure that no attention is paid to those tokens.

I think it may be due to we don't want to compute the loss of padding and the weight of the padding position should be $$0.$$

Answer is: we don't want softmax in attention to be affected by padded parts of sequences.

Sequences have different lengths:

• if sequence is too long, we trim it down
• if sequence is too shot, we pad remaining part with either tokens or values like 0

But these values in embeddings, further down the road, will affect attention's output, because of softmax: For example, if we have vector = [2, 0.5, 0.8, 1, 0, 0, 0, 0] (last 4 values are padded part), softmax output will be [0.41, 0.09, 0.12, 0.15, 0.06, 0.06, 0.06, 0.06], yet, we know that last four elements have no real value and shouldn't influence output.

Thus, we create mask and apply it before softmax, setting padded values to -inf or something like -1e9. For example, previous vector after replacement will look like this: [2, 0.5, 0.8, 1, -1e9, -1e9, -1e9, -1e9] and here's softmax output: [0.53, 0.12, 0.16, 0.19, 0, 0, 0, 0]

Just an example why people want to apply masks to encoders.

There're unsupervised language models pre-trained with an unidirectional mask, for example GPT. If we want to leverage this pre-trained language model to build a encoder-decoder based machine translation model, we might want to apply the unidirectional mask in the same fashion it's pre-trained with.