# Why does logistic regression produce well-calibrated models?

I understand that one of the reason logistic regression is frequently used for predicting click-through-rates on the web is that it produces well-calibrated models. Is there a good mathematical explanation for this?

• logistic regression made to predict probabilities --> which lead to calibrated predictions if not overfit. while most machine learning models do not predict probabilites, but rather a classes - and there is some contortion to derived pseudo-probabilites from these predictions - > hence note well calibrated Apr 23, 2016 at 5:10
• I should have clarified in the question, but my question was more about why it's the case that LR is so useful for predicting probabilities. Apr 23, 2016 at 16:19
• It's worth noting that you can simply fit a logistic regression to the output of a poorly-calibrated classifier to get a calibrated model. This is called Platt Scaling en.wikipedia.org/wiki/Platt_scaling May 23, 2019 at 12:19

Yes.

The predicted probability vector $p$ from logistic regression satisfies the matrix equation

$$X^t(p - y) = 0$$

Where $X$ is the design matrix and $y$ is the response vector. This can be viewed as a collection of linear equations, one arising from each column of the design matrix $X$.

Specializing to the intercept column (which is a row in the transposed matrix), the associated linear equation is

$$\sum_i( p_i - y_i) = 0$$

so the overall average predicted probability is equal to the average of the response.

More generally, for a binary feature column $x_{ij}$, the associated linear equation is

$$\sum_i x_{ij}(p_i - y_i) = \sum_{i \mid x_{ij} = 1}(p_i - y_i) = 0$$

so the sum (and hence average) of the predicted probabilities equals the sum of the response, even when specializing to those records for which $x_{ij} = 1$.

• @MatthewDrury how can I interpret your first equation? is $p$ of the form $1/(1+\exp(-x))$?Nevertheless this linear relation holds? Thank you! Feb 15, 2018 at 11:16
• Yes, p is of that form. The first equation comes from setting the derivative of the loss function to zero. Feb 15, 2018 at 15:49
• This addresses only calibration-in-the-large which is not what we want: calibration-in-the-small. Aug 20, 2018 at 12:35
• @FrankHarrell Care to elaborate? I haven't heard those terms before. Aug 20, 2018 at 16:43
• There's a long history in the probability forecast literature dating by to US Weather Service 1950 - that's where the Brier score was first use. Calibration-in-the-small means that if looked at predicted risks of 0.01, 0.02, ..., 0.99, each of these is accurate, i.e., for all the times when the predicted risk was 0.4, the outcome happened about 0.4 of the time. I call "calibration-in-the-tiny" the next step: for males in which the prediction was 0.4 was the outcome present 0.4 of the time, then for females. Aug 20, 2018 at 22:22

I think I can provide you an easy-to-understand explanation as follows:

We know that its loss function can be expressed as the following function:
$$J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[ y^{(i)}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\log\left(1-h_\theta \left(x^{(i)}\right)\right)\right]$$
Where m represents the number of all the training cases, $$y^{(i)}$$ the label of the ith case, $$h_{\theta}(x^{(i)})$$ the predicted probability of the ith case: $$\frac{1}{1+\exp[-\alpha -\sum_j \theta_j x^{(i)}_j]}$$. (notice the bias $$\alpha$$ here)

Since the goal of training is to minimize the loss function, let us evaluate its partial derivative with respect to each parameter $$\theta_j$$(the detailed derivation can be found here):
$$\frac{\partial J(\theta)}{\partial \theta_j}=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)}$$
And setting it to zero yeils:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_j^{(i)}=\sum_{i=1}^m y^{(i)}\,x_j^{(i)}$$

That means that if the model is fully trained, the predicted probabilities we get for the training set spread themselves out so that for each feature the sum of the weighted (all) values of that feature is equal to the sum of the values of that feature of the positive observations.

The above fits every feature so as the bias $$\alpha$$. Setting $$x_0$$ as 1 and $$\alpha$$ as $$\theta_0$$ yeilds:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_0^{(i)}=\sum_{i=1}^m y^{(i)}\,x_0^{(i)}$$ Then we get: $$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)=\sum_{i=1}^m y^{(i)}$$ Where $$h_\theta\left(x^{(i)}\right)$$ is the predicted probability of the fully trained model for the ith case. And we can write the function in a compact way: $$\sum_{i=1}^m p^{(i)} =\sum_{i=1}^m y^{(i)}$$

We can see obviously that the logistic regression is well-calibrated.

Logistic regression models are typically trained by minimizing the negative log likelihood. Maximum-likelihood models are known to asymptotically identify the solution minimizing the Kullback-Leibler divergence $$D_{KL}(P, Q)$$ between the empirical data distribution $$P$$ and the distribution $$Q$$ described by the model, see wiki. Minimizing the KL divergence between the two distributions implies optimizing for calibration.