# Premises

I'm training a convolutional neural network (ConvNet) on 51 subclasses in the ImageNet dataset. In order to keep an eye on overfitting, I have been suggested to plot training and testing loss function values (using a negative log-likelihood criterion) and accuracy (correct guesses over total number of sample). So far, I have obtained consistent results, similar to the example below.

Then, I killed the overfitting by using a dropout technique, producing the following chart.

# Question

Now I'm puzzled. I have comparable cross-entropy errors, but quite different accuracies. Shouldn't a similar loss function value, for train and test, give me similar precisions?

Rewording the question: why $e_{\mathrm{test}}\simeq e_{\mathrm{train}}\not\Rightarrow a_{\mathrm{test}}\simeq a_{\mathrm{train}}$ where $e$ stays for cross-entropy error (defined below) and $a$ stays for accuracy?

# Definitions

I've been using the terms cross-entropy errors and loss function values interchangeably. What I am referring at is the average prediction error which is given by

$$e = - \frac{1}{m}\sum_{i=1}^{m}{\ln[p(y_i|x_i)]}$$

where $m$ is the number of images in the training or testing sets, $y_i$ is the true label and $p(y_i|x_i)$ is the model's output probability associated to the input $x_i$ to be classified as $y_i$.

• As a sidenote, I had the opposite effect with dropout on MNIST. Empirical loss stayed constant at ~150 errors, but the NLL went up again. Weird. – bayerj Mar 12 '14 at 19:58
• May I ask you to define empirical loss? If the test cross-entropy rises, then you're overfitting. I would suggest you to open a new question and perhaps link it here :) – Atcold Mar 13 '14 at 20:33
• By empirical loss I mean the fraction of wrongly classified samples. – bayerj Mar 14 '14 at 8:57
• Yeah, as I said, you are overfitting. Apply dropout to every hidden layer (are you using ConvNets?) and do use a lower probability so that you squeeze all the juice out of your model. Furthermore, add dropout to the input space as well (but stay below .2), so that you become more robust to noise! :) – Atcold Mar 16 '14 at 17:55
• My point was not the overfitting but the divergence between the NLL and the empirical loss, which is the ultimate quantity of interest. I suppose the empirical loss on the validation set is a much better indicator for future classification generalization than the NLL. It depends on your use case of course (maybe a probabilistic model is what you want, after all.) – bayerj Mar 17 '14 at 8:59