I am trying to understand how encoder-decoder models works.

The encoder receives a sequence and the length is known. However the output of the encoder is just a single word vector capturing the sentence meaning. (if I understand it correctly)

Then this single vector is passed to the decoder and it somehow manages to extract single words from it.

Does the decoder simply chose a sequence of previously defined list of sentences, the training data, or it generates the sentence on the fly?

  • 1
    $\begingroup$ The decoder produces one softmax output vector at a time (corresponding to a single word/character/wordpiece) until it produces an End of Sentence token. Proper training just happens to lead to the "idea" of finite sentences being baked into the transformer. If you haven't read the Illustrated Transformer, I recommend it. $\endgroup$ – Alex L Feb 21 at 22:09
  • $\begingroup$ This article is indeed very good. It cleared things a lot. One thing I failed to understand is, how the decoder "subtracts" words from its current vector. By applying linear + softmax layers until reaching END token. How does the vector change during these operations and how is reachin the end token guaranteed. $\endgroup$ – Borislav Stoilov Feb 21 at 22:36
  • $\begingroup$ Really study that article, it's one of the most accessible. I'm not sure I follow your question. Are you talking about the loss function determining the predicted token's error? In regard to your last statement, AFAIK there is no proven "guarantee" that the statement will end. Incorrect length sequences are penalized though (arbitrary additional words would not correspond with the training sequences), and the positional encoding added in the encoder/decoder stacks help determine correct lengths. $\endgroup$ – Alex L Feb 21 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.