3
$\begingroup$

I've recently studied probability calibration and have investigated many examples and revisited old models of mine to find that they are all poorly calibrated. The idea of stacking also has been cast in a new light.

However intuitive the "calibration" process is, I find it hard to understand why even very simple models become so over-confident.

I'd be super grateful if you can share some insights into how this happens, as I would have guessed that many simpler (especially linear) models would reflect the true class probabilities.

Many thanks in advance!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Logistic regression models generally are fairly well-calibrated (on their training set, hence @sjw's answer); can you provide some specific examples? // You're not resampling, are you? $\endgroup$
    – Ben Reiniger
    Commented Dec 15, 2022 at 16:49
  • $\begingroup$ You describe a very general case and there might be a lot of causes. We do not know which one would answer your question. Could you add some more focus by giving an example of what situation you mean whera a model is being overconfident. $\endgroup$
    – Sextus Empiricus
    Commented Dec 15, 2022 at 17:29
  • $\begingroup$ @BenReiniger I did see them being generally better calibrated than other models such as decision trees, I just for some reason expected them to be near perfect, but redundancy as in sjw's answer make a lot of sense. I did resample decision trees and I did expect those to be biased since I a actively changing the distribution $\endgroup$
    – Oliver
    Commented Dec 15, 2022 at 17:49

1 Answer 1

Reset to default
2
$\begingroup$

Suppose there are for example many noise covariates and a very small sample. The linear model will by chance estimate nonzero coefficients on some of these (overfit). This can lead to higher confidence in the class probabilities.

If for example you have 2 variables and both are noise, if the model correctly estimates (doesn't overfit) by chance, you will get zero coefficients on both, and your estimated prob will be 0.5 each. Suppose you accidentally estimate a nonzero coefficient due to a spurious correlation between the response and a covariate (overfit); the estimated prob will then be greater or less than 0.5 (more confident).

This is a question of bias variance tradeoff, I expect, which varies by sample size, signal, etc. In other words, a linear model might be a complex model for a dataset without much signal; model complexity is relative, not absolute - a fact that is often overlooked in the applied literature.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks a lot, I think this makes a lot of sense. So for example, if I had a training set with 10 features but a lot of redundancy in these then the model would use the other features to overfit and this somehow leads to higher expected probabilities than that are actually there? $\endgroup$
    – Oliver
    Commented Dec 15, 2022 at 16:19
  • 1
    $\begingroup$ Yes edited a bit $\endgroup$
    – Sam Weisenthal
    Commented Dec 15, 2022 at 16:47
  • $\begingroup$ Thanks a million, this is an intuitive answer! $\endgroup$
    – Oliver
    Commented Dec 15, 2022 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.