1
$\begingroup$

In the paper 'Attention is all you need' the model architecture of The Transformer is described. Both in the encoder as well as in the decoder, there is a feed forward network. If I understant it correctly, the input to this feed forward network is the embedding/ continuous representation with similar dimensions.

However, who can tell me what the target variable is on which the neural network is trained?

$\endgroup$
1
$\begingroup$

It doesn't have or need a target variable, because it is just used as a component of the larger transformer model.

When the transformer model is trained (on say a translation task, where the target variable is the target sentence), all the parameters, including those of the feed-foward network will also be trained.

$\endgroup$
1
  • $\begingroup$ when u say trained on the feed forward network, is it only on the decoder linear layer or on all the feed forward layer the tranformer consists? I mean incl. the encoder and decoder feed forward layers? $\endgroup$ – prog Jan 8 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.