I am trying to reproduce a neural probabilistic language model described by Bengia (without distributed softmax computation).
According to Bengio et al. "training is achieved by looking for model parameters that maximize the training corpus log-likelihood". However, I can't find any mentions regarding what is actually used as a target value.
When building a neural network for classifying handwritten digits from a MNIST dataset, we may use a matrix (or a vector) of normalized values that represent corresponding pixels' brightness levels as input X. And one-hot vector of 10 elements may be used to represent an output Y (with all elements set to 0 except one set to 1 which index represents a digit label itself). This model is clear.
In case of a neural probabilistic language model we use context words' indices as input. These indices are then transformed to a set of feature vectors using vocabulary. Eventually, after processing by a hidden layer and softmax function we get a list of probabilities for each word of a vocabulary. And here I stuck. In a description of a backward/update phase Bengio uses:
∂L/∂yj ← 1j==wt − pj
to calculate initial gradient, where pj is a probability of j-th word in a vocabulary. I can't comprehend this part. Where does "1" come from? Is it necessary? Can we use the same approach to represent a target probability vector as in case with MNIST?