Going through the famous notebook: Annotated transformers to be found here, I noticed that during the training, the model is fed the target sentence.

If the model is fed the target sentence from the get go, what would the model be learning ?


A transformer is a generative model. The goal of all generative models is to learn the probability distribution of the data (in this case, natural language sentences).

To say "the model is fed the target sentence" isn't technically wrong, but it can be confusing. Instead, think of a generative model as a box with the following abilities*:

  1. You can give it a piece of data (a sentence) and ask how probable that sentence is.
  2. You can ask it to give you a sentence randomly sampled from the distribution it models.

Some special types of generative models, including transformers, also have the following property:

  1. You can sample certain conditionals of the modeled distribution -- for example, sampling the rest of a sentence given the first few words.

Some tutorials on natural language models talk about "predicting the next word" -- what this refers to is just (3). While it's an easy way to think about it -- especially coming from supervised models -- it often creates confusion if you try to draw too many analogies between this and supervised learning. I would focus more on understanding how the model does (1) aka "training" and (2) aka "inference" or "sampling".

*Of course sampling or computing the (normalized) probability is not always tractable or cheap. Though at least an approximation or proxy is often used.

  • $\begingroup$ So what does the model actually learns during training ? And how can it predict the next word ? $\endgroup$
    – user297904
    Oct 11 '20 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.