Why do we feed the target sentence to the model during transformers training? Going through the famous notebook: Annotated transformers to be found here, I noticed that during the training, the model is fed the target sentence.
If the model is fed the target sentence from the get go, what would the model be learning ?
 A: A transformer is a generative model. The goal of all generative models is to learn the probability distribution of the data (in this case, natural language sentences).
To say "the model is fed the target sentence" isn't technically wrong, but it can be confusing. Instead, think of a generative model as a box with the following abilities*:

*

*You can give it a piece of data (a sentence) and ask how probable that sentence is.

*You can ask it to give you a sentence randomly sampled from the distribution it models.

Some special types of generative models, including transformers, also have the following property:


*You can sample certain conditionals of the modeled distribution -- for example, sampling the rest of a sentence given the first few words.

Some tutorials on natural language models talk about "predicting the next word" -- what this refers to is just (3). While it's an easy way to think about it -- especially coming from supervised models -- it often creates confusion if you try to draw too many analogies between this and supervised learning. I would focus more on understanding how the model does (1) aka "training" and (2) aka "inference" or "sampling".

*Of course sampling or computing the (normalized) probability is not always tractable or cheap. Though at least an approximation or proxy is often used.
