Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the true probabilities of class membership.

However, I have read on Cross Validated and perhaps elsewhere that neural networks are notorious for being overly confident. That is, they will be happy to predict something like $P(1) = 0.9$ when they should be predicting $P(1) = 0.7$, which still favors class $1$ over class $0$ but by less.

If neural networks are optimizing a proper scoring rule like crossentropy loss, how can this be?

All that comes to mind is that the model development steps optimize improper metrics like accuracy. Sure, the model in cross validation is fitted to the training data using crossentropy loss, but the hyperparameters are tuned to get the highest out-of-sample accuracy, not the lowest crossentropy loss.

(But then I figure that the model would be less confident in its predictions. Why be confident in your prediction when you get the right classification with a low-confidence classification like $0.7$ than a high-confidence classification like $0.9$?)

  • 4
    $\begingroup$ Good question. I suspect part of the answer is that you can overfit to proper scoring rules just as easily as to other KPIs if you use them in-sample. After all, OLS is fitted by maximizing the log likelihood, which is the log score, a proper scoring rule - but that OLS can overfit is common knowledge. $\endgroup$ Jun 30 at 14:38
  • $\begingroup$ I think the key term to google is Expected Callibration Error (ECE). I suspect this post will answer your question alondaks.com/2017/12/31/… $\endgroup$ Jun 30 at 15:20
  • $\begingroup$ @StephanKolassa Why would that be so unique to neural networks and not logistic regression? Is it a matter of a neural network having (perhaps) millions of parameters but the logistic regression maybe having dozens? $\endgroup$
    – Dave
    Jul 2 at 16:48
  • 1
    $\begingroup$ @Dave: yes, that makes sense. Logistic regression can also overfit if you over-parameterize it. And conversely, I would not expect a simple network architecture to overfit badly. $\endgroup$ Jul 2 at 19:25
  • $\begingroup$ @StephanKolassa indeed it can. LR can even overfit when it is not over-parameterised, which is why regularised (ridge) logistic regression is a very useful tool to have in your statistic toolbox. $\endgroup$ Jul 26 at 7:10

"If neural networks are optimizing a proper scoring rule like cross-entropy loss, how can this be?"

This is likely to be traditional over-fitting of the training data. A deep neural network can implement any mapping that a radial basis function neural network can implement (they are both universal aproximators). Consider a problem with a small data set and a narrow width for the Gaussian radial basis functions. It is possible that you might be able to place a basis function directly over each positive pattern, such that the value has decreased to nearly zero by the time you get to the nearest negative pattern. This model will give a probability of class membership of essentially zero or one for every training pattern (probably way over-confident) and a training set cross-entropy of zero. This means there will also be a zero cross-entropy solution for a suitably large deep neural network as well (the good thing is that solution is a lot harder to find for a DNN - sometimes local minima are a good thing).

Making architecture or hyper-parameter choices gives more ways in which to over-fit the data, but I suspect the largest part of the problem is traditional over-fitting of the training set, unless steps are taken to avoid it.

BTW using cross-entropy as the model selection criterion for tuning the model is not without it's own problems, for instance if you have one very confident miss-classification, then the entire cross-entropy is dominated by the contribution of that one test example. Something a little less sensitive, like the Brier score might be better (if less satisfying).

  • $\begingroup$ This is interesting and gets a +1 from me, but then why would the accuracy (or something similar) be decent out-of-sample? $\endgroup$
    – Dave
    Jul 26 at 10:00
  • $\begingroup$ I wouldn't recommend using accuracy for model selection either (because it is brittle). The problem is that you can have decent (but not perfect) out-of-sample accuracy, but the model be very confident in one of its out-of-sample miss-classifications, which results in an essentially infinitely bad out-of-sample cross-entropy. There are ways of getting round this, such as placing a limit on the plausible confidence of the model, but something like Brier score may be easier. $\endgroup$ Jul 26 at 10:09
  • $\begingroup$ I use least-square support vector machines a fair bit, and it seems consistent to use least-squares error on the out-of-sample criterion used for model selection as it is also used in-sample for fitting the model. It would be nice to use out-of-sample cross-entropy for model selection for kernel logistic regression, but you run in to the problem I mentioned earlier in some cases. $\endgroup$ Jul 26 at 10:11
  • $\begingroup$ BTW I did a bit of a comparative study on this sort of thing for LSSVMs for a machine learning challenge at a conference ieeexplore.ieee.org/abstract/document/1716307 (pre-print here: theoval.cmp.uea.ac.uk/publications/pdf/ijcnn2006a.pdf ). Looks like least-squares based model selection criterion (PRESS) was a good approach. $\endgroup$ Jul 26 at 10:19
  • $\begingroup$ It’s the infinitely-bad crossentropy that’s tripping me up. If an developer is optimizing an out-of-sample metric like accuracy (probably even AUC), one bad miss like that should get washed out by the other good predictions, but the out-of-sample crossentropy loss will be poor. That should alert the modeler that something weird has happened…which gets missed by looking at the threshold-based metrics. $\endgroup$
    – Dave
    Jul 26 at 10:27

There are some interesting properties about cross entropy loss (well, even logistic loss).

Not only we want to classify the instance correctly, we want strong options

Please check following example, note, weakly and correctly classifying one data point is almost as bad ad wrongly classifying the point.

  import numpy as np

  def cross_entropy(pred_prob, target): 
      return -np.sum(np.log(pred_prob) * (target))

  target = np.array([1,0,0])

  pred_prob = np.array([1/3,1/3,1/3])
  print('3 classes, even dist\t',cross_entropy(pred_prob, target))

  pred_prob = np.array([0.3334,0.3333,0.3333])
  print('3 classes, weakly right \t',cross_entropy(pred_prob, target))

  pred_prob = np.array([0.3333,0.3333,0.3334])
  print('3 classes, weakly wrong \t',cross_entropy(pred_prob, target))

I also had a related question here

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

In addition, the as discussed in another answer, one major reason model is confidently wrong is the overfitting.

I recently had some interesting ideas on what may be happening inside DNN. I think essentially the overfitted model is trying to learn some "hash functions" and hardly remember "the has and the target".

In this way, it is easily to get very low loss in training data. And this "hash" is specific to certain training examples, and will very likely to be over confident and have very strong option on one class.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.