I am trying to understand the transformer model from Attention is all you need, following the annotated transformer.

The architecture looks like this:


Everything is essentially clear, save for the output embedding on the bottom right. While training, I understand that one can use the actual target as input - all one needs is to

  • shift the target by one position to the right
  • use a mask to prevent using - say - the $n+k$-th word from the output to learn the $n$-th one

What is not clear to me is how to use the model at inference time. When doing inference, one of course does not have the output - what goes there?


1 Answer 1


A popular method for such sequence generation tasks is beam search. It keeps a number of K best sequences generated so far as the "output" sequences.

In the original paper different beam sizes was used for different tasks. If we use a beam size K=1, it becomes the greedy method in the blog you mentioned.

  • $\begingroup$ Yeah, I saw it mentioned in the blog post, but without much explanation. If I understand correctly, the greedy method amounts to starting from any random output (all 1 seems to be used in the post), use that to generate the first token, then repeat with the first token to generate the second token and so on. If so, it seems that - unlike in training - decoding in inference happens in a loop, one token at a time, not unlike from what happens with RNN. Am I understanding correctly? $\endgroup$ Commented Nov 12, 2018 at 16:37
  • $\begingroup$ Beam search would amount to using the same strategy, but keeping the best k tokens at each step, if I understand correctly. Again, while training can exploit parallelization, it seems that decoding would have to happen serially. Please, tell me if I am missing something $\endgroup$ Commented Nov 12, 2018 at 16:39
  • $\begingroup$ @AndreaFerretti yes! it has to be done step by step, for generating the first token, we usually initiate the output with a fixed start_symbol, like all ones. $\endgroup$
    – dontloo
    Commented Nov 13, 2018 at 2:40
  • $\begingroup$ Thank you, everything is clear. I was mislead by the remarks I heard that the transformer is more hardware friendly than usual RNN architectures, due to the fact that it avoids the sequential loop, and in fact that works at training time, just not for inference $\endgroup$ Commented Nov 13, 2018 at 8:52
  • 1
    $\begingroup$ @jonathanking hi by decoding I meant to generate sequences, by evaluation I meant to test the probability of a given sequence, you may take a look at how it's defined in the context of HMMs jedlik.phy.bme.hu/~gerjanos/HMM/node6.html $\endgroup$
    – dontloo
    Commented Sep 25, 2019 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.