Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.
Why is logistic regression well calibrated? How could one ruin the calibration of a logistic regression (not that one would ever want to - just as an exercise)?
Although this question and its first answer seems to be focused on theoretical issues of logistic regression model calibration, the issue of:
How could one ruin the calibration of a logistic regression...?
deserves some attention with respect to real-world applications, for future readers of this page. We shouldn't forget that the logistic regression model has to be well specified, and that this issue can be particularly troublesome for logistic regression.
First, if the log-odds of class membership is not linearly related to the predictors included in the model then it will not be well calibrated. Harrell's chapter 10 on Binary Logistic Regression devotes about 20 pages to "Assessment of Model Fit" so that one can take advantage of the "asymptotic unbiasedness of the maximum likelihood estimator," as @whuber put it, in practice.
Second, model specification is a particular issue in logistic regression, as it has an inherent omitted variable bias that can be surprising to those with a background in ordinary linear regression. As that page puts it:
Omitted variables will bias the coefficients on included variables even if the omitted variables are uncorrelated with the included variables.
That page also has a useful explanation of why this behavior is to be expected, with a theoretical explanation for related, analytically tractable, probit models. So unless you know that you have included all predictors related to class membership, you might run into dangers of misspecification and poor calibration in practice.
With respect to model specification, it's quite possible that tree-based methods like random forest, which do not assume linearity over an entire range of predictor values and inherently provide the possibility of finding and including interactions among predictors, will end up with a better-calibrated model in practice than a logistic regression model that does not take interaction terms or non-linearity sufficiently into account. With respect to omitted-variable bias, it's not clear to me whether any method for evaluating class-membership probabilities can deal with that issue adequately.
Logistic regression is a classification method that basically learns a probability function $\pi_\theta(x)$ over the input space by fitting the parameters $\theta$. If the predicted probabilities are learned with the appropriate loss function, than logistic regression has the potential to learn an unbiased estimation of the binary event probabilities, whenever its has sufficient capacity (input features).
The log loss allows such unbiased estimation. Consider the fact that the log loss function is simply the negative log likelihood of a Bernoulli distribution $z \thicksim \text{Ber}(p)$. The maximum likelihood estimation for $p$ is unbiased given a set of observations for variable $z$. In the case of classification over some input space $\mathcal{X}$, one can imagine having one Bernoulli distribution for all points in $\mathcal{X}$. Most often, you will only have 1 observation $y_i$ per Bernoulli distribution, which is located at $x_i$. Jointly applying maximum likelihood estimation for all observed Bernoulli distributions $y_i \thicksim \text{Ber}(\pi(x_i))$ will apply several constraints to $\pi_\theta$. Since all these constraints leads to unbiased estimations, and as long as the function $\pi_\theta$ is sufficently flexible to fit the true underlying probability function $\pi^*$, then the learning procedure is consistent and will converge to the optimal model as you get more data. Thus, limiting the model capacity (fewer features for instance) can hinder the calibration of a logistic regression by increasing the distance between the best learnable model and the true model.
Using an incorrect observation model with the logistic regression will lead to uncalibrated probabilities. Modeling binary events with a normal distribution is inappropriate, and should not be used in combination with logistic regression. The loss function corresponding to the normal distribution observation model is the Mean Squared Error. Thus, using an MSE loss would certaintly hinder its calibration.
