Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)

Question

I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any context word) in a sequence of words.

However, when reading Bengio et al's original Neural LM paper (a neural probabilistic language model) it isn't very clear to me how they built the objective function for the optimizer.

From the paper:

Does this mean that the authors had to build the target by manually counting the empirical conditional probability for every "row" in the training set?

For instance, say the network is processing the string "foo bar baz" right now:

The target for this self-supervision instance would need to be the number of times "baz" followed "foo bar" divided by all occurrences of "foo bar" in the validation set?

Answer 1

It is actually the same cross-entropy as with current language models; only the notation is different. It is well illustrated in Figure 1 of the paper:

Function $f$ already returns the $i$-th index of the final softmax. So, calling $f(w_t, w_{t-1}, \ldots, w_{t-n+1};\theta)$ actually means: take the embeddings of words $w_{t-1}, \ldots, w_{t-n+1}$ (denoted as function $C$), apply the non-linear layer and the softmax layer and take the predicted probability corresponding to $w_t$. The loss function can be therefore written as $\sum_t \log P (w_t|w_{t-1}, \ldots, w_{t-n+1})$, which is the log-likelihood. Today's convention is to use negative log-likelihood.