Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder this is not possible (in both training and testing), as self attention is calculated based on previous timestep outputs. Even if we consider some technique like teacher forcing, where we are concatenating expected output with obtained, this still has a sequential input from the previous timestep. In this case, apart from the improvement in capturing long-term dependencies, is using a transformer-decoder better than say an lstm when comparing purely on the basis of parallelization?
2
-
$\begingroup$ Since you seem to be referring to some particular neural network architecture, could you please provide us with the reference describing this network? $\endgroup$– TimCommented May 23, 2019 at 18:39
-
$\begingroup$ arxiv.org/pdf/1706.03762.pdf This is the original transformer paper, and the model I am referring to is given in Figure 1. $\endgroup$– shiredude95Commented May 23, 2019 at 19:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In the standard Transformer model as introduced by Vaswani et al. it is not possible, because generating a word is always conditioned on the previously decoded words, so there is no other option than generating words one by one.
Recently, there appeared several papers on so-called non-autoregressive models which parallelize the decoder as well, but the problem is that the words "do not know" what the previous words are, so there is a big problem with fluency. Another
problem is that you need to somehow estimate how long the target sentence will be
because you cannot wait until a </s> symbol.
There were several papers dealing with that:
Gu et al., 2017 use latent variables for fertility in the middle, but they have to sample from the model and re-score the outputs with the standard Transformer to get reasonable results.
Lee et al. (2018) use a simple classifier to estimate target sentence length. They also have a second non-autoregressive decoder that is used for iterative refinement of the generated sentence and gain better fluency.
Libovický and Helcl (2018) used an ASR-like approach and avoided the target length estimation using CTC loss.
Most recently, Ghazvininejad et al. (2019) used BERT-like training. There is also a blog post summarizing it.
