I am trying to reproduce a neural probabilistic language model described by Bengia (without distributed softmax computation).

According to Bengio et al. "training is achieved by looking for model parameters that maximize the training corpus log-likelihood". However, I can't find any mentions regarding what is actually used as a target value.

When building a neural network for classifying handwritten digits from a MNIST dataset, we may use a matrix (or a vector) of normalized values that represent corresponding pixels' brightness levels as input X. And one-hot vector of 10 elements may be used to represent an output Y (with all elements set to 0 except one set to 1 which index represents a digit label itself). This model is clear.

In case of a neural probabilistic language model we use context words' indices as input. These indices are then transformed to a set of feature vectors using vocabulary. Eventually, after processing by a hidden layer and softmax function we get a list of probabilities for each word of a vocabulary. And here I stuck. In a description of a backward/update phase Bengio uses:

∂L/∂yj ← 1j==wt − pj

to calculate initial gradient, where pj is a probability of j-th word in a vocabulary. I can't comprehend this part. Where does "1" come from? Is it necessary? Can we use the same approach to represent a target probability vector as in case with MNIST?

| cite | improve this question | | | | |

I'm sure you already figured this out by now but for future reference, I'm going to answer anyway.

I think the confusing part in the term $\frac{\partial L}{\partial y_j}=1_{j=w_t}-p_{j}$ is the indicator function $1_{j=w_{t}}$ which is defined as $1$ when $j=w_{t}$ and $0$ otherwise. Therefore, in order to minimize the loss you want to obtain predictions $p_{j}$ that are close to $1$ for the output unit corresponding to the correct word $w_{t}$ and values near $0$ for the other output units.

As you can see, this is very similar to the training procedure that you described in the MNST dataset.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.