8
$\begingroup$

Premises

I'm training a convolutional neural network (ConvNet) on 51 subclasses in the ImageNet dataset. In order to keep an eye on overfitting, I have been suggested to plot training and testing loss function values (using a negative log-likelihood criterion) and accuracy (correct guesses over total number of sample). So far, I have obtained consistent results, similar to the example below.

Clear overfitting

Then, I killed the overfitting by using a dropout technique, producing the following chart.

Killed overfitting with dropout

Question

Now I'm puzzled. I have comparable cross-entropy errors, but quite different accuracies. Shouldn't a similar loss function value, for train and test, give me similar precisions?

Rewording the question: why $e_{\mathrm{test}}\simeq e_{\mathrm{train}}\not\Rightarrow a_{\mathrm{test}}\simeq a_{\mathrm{train}}$ where $e$ stays for cross-entropy error (defined below) and $a$ stays for accuracy?

Definitions

I've been using the terms cross-entropy errors and loss function values interchangeably. What I am referring at is the average prediction error which is given by

$$e = - \frac{1}{m}\sum_{i=1}^{m}{\ln[p(y_i|x_i)]}$$

where $m$ is the number of images in the training or testing sets, $y_i$ is the true label and $p(y_i|x_i)$ is the model's output probability associated to the input $x_i$ to be classified as $y_i$.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ As a sidenote, I had the opposite effect with dropout on MNIST. Empirical loss stayed constant at ~150 errors, but the NLL went up again. Weird. $\endgroup$
    – bayerj
    Mar 12, 2014 at 19:58
  • $\begingroup$ May I ask you to define empirical loss? If the test cross-entropy rises, then you're overfitting. I would suggest you to open a new question and perhaps link it here :) $\endgroup$
    – Atcold
    Mar 13, 2014 at 20:33
  • $\begingroup$ By empirical loss I mean the fraction of wrongly classified samples. $\endgroup$
    – bayerj
    Mar 14, 2014 at 8:57
  • $\begingroup$ Yeah, as I said, you are overfitting. Apply dropout to every hidden layer (are you using ConvNets?) and do use a lower probability so that you squeeze all the juice out of your model. Furthermore, add dropout to the input space as well (but stay below .2), so that you become more robust to noise! :) $\endgroup$
    – Atcold
    Mar 16, 2014 at 17:55
  • $\begingroup$ My point was not the overfitting but the divergence between the NLL and the empirical loss, which is the ultimate quantity of interest. I suppose the empirical loss on the validation set is a much better indicator for future classification generalization than the NLL. It depends on your use case of course (maybe a probabilistic model is what you want, after all.) $\endgroup$
    – bayerj
    Mar 17, 2014 at 8:59

1 Answer 1

Reset to default
1
$\begingroup$

It is suggested by Prof. Frank Harrell in my post that accuracy in percentage is an improper scoring rule in which the accuracy score is possibly optimized by the wrong model, and addition of a highly important predictor may cause the model less accurate. Besides, the accuracy scoring rule is high variance. On the other hand, the log-likelihood is considered as a proper scoring rule.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Your answer is quite confusing. I'm not sure I understand what you're saying. Anyhow, my question was why similar cross-entropy error have different accuracy for training and testing set. Accuracy, or better, #correct guesses over #total samples is a pretty solid figure, i.e. reaching 100% tells me I can predict the whole testing set labels given its images. Perhaps you have answered my question, but I haven't got it. Could you please reword it? $\endgroup$
    – Atcold
    Mar 7, 2014 at 4:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.