9
$\begingroup$

In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.

Why is logistic regression well calibrated? How could one ruin the calibration of a logistic regression (not that one would ever want to - just as an exercise)?

$\endgroup$
4
$\begingroup$

Although this question and its first answer seems to be focused on theoretical issues of logistic regression model calibration, the issue of:

How could one ruin the calibration of a logistic regression...?

deserves some attention with respect to real-world applications, for future readers of this page. We shouldn't forget that the logistic regression model has to be well specified, and that this issue can be particularly troublesome for logistic regression.

First, if the log-odds of class membership is not linearly related to the predictors included in the model then it will not be well calibrated. Harrell's chapter 10 on Binary Logistic Regression devotes about 20 pages to "Assessment of Model Fit" so that one can take advantage of the "asymptotic unbiasedness of the maximum likelihood estimator," as @whuber put it, in practice.

Second, model specification is a particular issue in logistic regression, as it has an inherent omitted variable bias that can be surprising to those with a background in ordinary linear regression. As that page puts it:

Omitted variables will bias the coefficients on included variables even if the omitted variables are uncorrelated with the included variables.

That page also has a useful explanation of why this behavior is to be expected, with a theoretical explanation for related, analytically tractable, probit models. So unless you know that you have included all predictors related to class membership, you might run into dangers of misspecification and poor calibration in practice.

With respect to model specification, it's quite possible that tree-based methods like random forest, which do not assume linearity over an entire range of predictor values and inherently provide the possibility of finding and including interactions among predictors, will end up with a better-calibrated model in practice than a logistic regression model that does not take interaction terms or non-linearity sufficiently into account. With respect to omitted-variable bias, it's not clear to me whether any method for evaluating class-membership probabilities can deal with that issue adequately.

$\endgroup$
5
$\begingroup$

Logistic regression is a classification method that basically learns a probability function $\pi_\theta(x)$ over the input space by fitting the parameters $\theta$. If the predicted probabilities are learned with the appropriate loss function, than logistic regression has the potential to learn an unbiased estimation of the binary event probabilities, whenever its has sufficient capacity (input features).

The log loss allows such unbiased estimation. Consider the fact that the log loss function is simply the negative log likelihood of a Bernoulli distribution $z \thicksim \text{Ber}(p)$. The maximum likelihood estimation for $p$ is unbiased given a set of observations for variable $z$. In the case of classification over some input space $\mathcal{X}$, one can imagine having one Bernoulli distribution for all points in $\mathcal{X}$. Most often, you will only have 1 observation $y_i$ per Bernoulli distribution, which is located at $x_i$. Jointly applying maximum likelihood estimation for all observed Bernoulli distributions $y_i \thicksim \text{Ber}(\pi(x_i))$ will apply several constraints to $\pi_\theta$. Since all these constraints leads to unbiased estimations, and as long as the function $\pi_\theta$ is sufficently flexible to fit the true underlying probability function $\pi^*$, then the learning procedure is consistent and will converge to the optimal model as you get more data. Thus, limiting the model capacity (fewer features for instance) can hinder the calibration of a logistic regression by increasing the distance between the best learnable model and the true model.

Using an incorrect observation model with the logistic regression will lead to uncalibrated probabilities. Modeling binary events with a normal distribution is inappropriate, and should not be used in combination with logistic regression. The loss function corresponding to the normal distribution observation model is the Mean Squared Error. Thus, using an MSE loss would certaintly hinder its calibration.

$\endgroup$
  • 2
    $\begingroup$ Careful calling logistic regression a classification method on this site! Thank you for the answer - it seems you are implying that the log loss objective is the reason for calibration (assuming the model is adequately flexible)? $\endgroup$ – user0 Feb 3 at 2:03
  • 1
    $\begingroup$ A follow up - you say calibration requires unbiased estimation of the probability - hence penalization ruins calibration? $\endgroup$ – user0 Feb 3 at 2:14
  • $\begingroup$ « LogisticRegression returns well calibrated predictions by default as it directly optimizes log-loss » - scikit-learn.org/stable/modules/calibration.html $\endgroup$ – cortax Feb 3 at 3:39
  • $\begingroup$ By definition, penalization or regularization, is a bias injection which often seeks to reduce the variance of the estimator. A massive regularization can dominate the data part of the objective function, and definitely ruin calibration. $\endgroup$ – cortax Feb 3 at 3:45
  • 2
    $\begingroup$ The scikit-learn quotation about "optimizes log loss" isn't an effective explanation, because there is no necessary connection between this and being unbiased. Unless I'm mistaken, the correct answer to the question will need to invoke the asymptotic unbiasedness of the maximum likelihood estimator typically used in logistic regression procedures. $\endgroup$ – whuber Feb 3 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.