Calculating gradient in a neural probabilistic language model

Question

I am trying to reproduce a neural probabilistic language model described by Bengia (without distributed softmax computation).

According to Bengio et al. "training is achieved by looking for model parameters that maximize the training corpus log-likelihood". However, I can't find any mentions regarding what is actually used as a target value.

When building a neural network for classifying handwritten digits from a MNIST dataset, we may use a matrix (or a vector) of normalized values that represent corresponding pixels' brightness levels as input X. And one-hot vector of 10 elements may be used to represent an output Y (with all elements set to 0 except one set to 1 which index represents a digit label itself). This model is clear.

In case of a neural probabilistic language model we use context words' indices as input. These indices are then transformed to a set of feature vectors using vocabulary. Eventually, after processing by a hidden layer and softmax function we get a list of probabilities for each word of a vocabulary. And here I stuck. In a description of a backward/update phase Bengio uses:

∂L/∂yj ← 1_j==wt − p_j

to calculate initial gradient, where p_j is a probability of j-th word in a vocabulary. I can't comprehend this part. Where does "1" come from? Is it necessary? Can we use the same approach to represent a target probability vector as in case with MNIST?

Answer 1

I'm sure you already figured this out by now but for future reference, I'm going to answer anyway.

I think the confusing part in the term $\frac{\partial L}{\partial y_j}=1_{j=w_t}-p_{j}$ is the indicator function $1_{j=w_{t}}$ which is defined as $1$ when $j=w_{t}$ and $0$ otherwise. Therefore, in order to minimize the loss you want to obtain predictions $p_{j}$ that are close to $1$ for the output unit corresponding to the correct word $w_{t}$ and values near $0$ for the other output units.

As you can see, this is very similar to the training procedure that you described in the MNST dataset.