How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?

Question

Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the true probabilities of class membership.

However, I have read on Cross Validated and perhaps elsewhere that neural networks are notorious for being overly confident. That is, they will be happy to predict something like $P(1) = 0.9$ when they should be predicting $P(1) = 0.7$, which still favors class $1$ over class $0$ but by less.

If neural networks are optimizing a proper scoring rule like crossentropy loss, how can this be?

All that comes to mind is that the model development steps optimize improper metrics like accuracy. Sure, the model in cross validation is fitted to the training data using crossentropy loss, but the hyperparameters are tuned to get the highest out-of-sample accuracy, not the lowest crossentropy loss.

(But then I figure that the model would be less confident in its predictions. Why be confident in your prediction when you get the right classification with a low-confidence classification like $0.7$ than a high-confidence classification like $0.9$?)

Answer 1

"If neural networks are optimizing a proper scoring rule like cross-entropy loss, how can this be?"

This is likely to be traditional over-fitting of the training data. A deep neural network can implement any mapping that a radial basis function neural network can implement (they are both universal aproximators). Consider a problem with a small data set and a narrow width for the Gaussian radial basis functions. It is possible that you might be able to place a basis function directly over each positive pattern, such that the value has decreased to nearly zero by the time you get to the nearest negative pattern. This model will give a probability of class membership of essentially zero or one for every training pattern (probably way over-confident) and a training set cross-entropy of zero. This means there will also be a zero cross-entropy solution for a suitably large deep neural network as well (the good thing is that solution is a lot harder to find for a DNN - sometimes local minima are a good thing).

Making architecture or hyper-parameter choices gives more ways in which to over-fit the data, but I suspect the largest part of the problem is traditional over-fitting of the training set, unless steps are taken to avoid it.

BTW using cross-entropy as the model selection criterion for tuning the model is not without it's own problems, for instance if you have one very confident miss-classification, then the entire cross-entropy is dominated by the contribution of that one test example. Something a little less sensitive, like the Brier score might be better (if less satisfying).

Answer 2

There are some interesting properties about cross entropy loss (well, even logistic loss).

Not only we want to classify the instance correctly, we want strong options

Please check following example, note, weakly and correctly classifying one data point is almost as bad ad wrongly classifying the point.

  import numpy as np

  def cross_entropy(pred_prob, target): 
      return -np.sum(np.log(pred_prob) * (target))

  target = np.array([1,0,0])

  pred_prob = np.array([1/3,1/3,1/3])
  print('3 classes, even dist\t',cross_entropy(pred_prob, target))

  pred_prob = np.array([0.3334,0.3333,0.3333])
  print('3 classes, weakly right \t',cross_entropy(pred_prob, target))

  pred_prob = np.array([0.3333,0.3333,0.3334])
  print('3 classes, weakly wrong \t',cross_entropy(pred_prob, target))

I also had a related question here

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

In addition, the as discussed in another answer, one major reason model is confidently wrong is the overfitting.

I recently had some interesting ideas on what may be happening inside DNN. I think essentially the overfitted model is trying to learn some "hash functions" and hardly remember "the has and the target".

In this way, it is easily to get very low loss in training data. And this "hash" is specific to certain training examples, and will very likely to be over confident and have very strong option on one class.

Answer 3

Over-confidence and calibration is a notorious open problem and challenge for neural networks. Neural networks are well-known for being over-confident. There are many research papers that have attempted to construct ways to improve calibration, with partial improvements, but the end result is that even with these improvements, the networks often remain over-confident.

I am not sure that anyone really knows for sure why this happens, or how to fix it. I have seen multiple theories, but my impression is that none seem fully convincing.

I would imagine that the very high number of parameters in neural networks may be relevant, but I'm not sure it is simply traditional overfitting. Normal engineering practice with training networks is to use regularization, particularly early stopping, and empirically this is effective at reducing overfitting; the gap between performance on the training set and performance on the validation set is often not too large. Many intuitions that we've built up on older statistical models (logistic regression and the sort) don't necessarily translate to modern neural networks.

I very much doubt that the problem is due to using non-proper scoring rules for model tuning. Over-confidence occurs even if we don't perform any model tuning.

It sounds like you are expecting that if we use a proper scoring rule, then we are guaranteed that the resulting model will be well-calibrated. I don't think there is any such guarantee, at least not for concrete problem instances. Standard arguments for using proper scoring rules talk about the asymptotics (optimizing to find the global minimum), but when training neural networks, we don't do that; we use early stopping.

I wonder if covariate shift might be part of the answer. In many settings, neural networks work very well on the exact data distribution they were trained on, but are very fragile: if the data distribution is slightly different than what they were trained on, their performance can deteriorate severely. This can lead to so-called "strong but wrong" predictions on out-of-distribution data, i.e., on instances that fall outside of the training distribution, the neural network's classification predictions are often wrong but it is highly confident in its wrong answer.

I suspect another possible perspective is that in practice people use all sorts of techniques to reduce overfitting in neural networks: early stopping, data augmentation, and more. Once you use these techniques as part of training, they can be viewed as effectively modifying the loss function with some additional penalty, and then the loss function is likely no longer a proper scoring rule due to this modification. So those guarantees might not apply. Yet from a practical, engineering standpoint, those techniques are absolutely essential to make neural networks work well at classification. (See also When is it appropriate to use an improper scoring rule?)