Attention is all you need: During training, does decoder generates the whole sentence in 1 shot and not sequentially word by word?

Question

In the famous paper "Attention is all you need", does the decoder generates the whole output sentence in one shot in parallel. In this case will the final softmax output the probabilities for each word for each position like this:

$$\left[(\text{pos 1}: P(w_1), P(w_2), \ldots, P(w_V)), (\text{pos 2}: P(w_1), P(w_2),... P(w_V)), \ldots, (\text{pos $N$}: P(w_1), P(w_2), \ldots, P(w_V) )\right]$$

where $N$ is the sequence length and $V$ the vocab size. Hence, are there $N$ sets of outputs from the softmax ?

Answer 1

No, the decoder cannot generate the outputs in parallel. It generates the output autoregressively, i.e., the probability distribution of the $i+1$-th token is conditioned on the $i$-th output token.

At the training time, this is simulated by the triangle maks in the self-attention that prevents the decoder from attending to the tokens that will be generated in the future (this is discussed in the Decoder paragraph in Section 3.1 of the paper). At the inference time, the triangle maks is not needed because the future words simply are not there.

Methods for generating all outputs parallel exist. This is called non-autoregressive machine translation, but in this case, you need to deal with the fact output tokens become conditionally independent. This can be solved e.g. using the CTC loss or by iterative masking and improving the output.