Why does logistic regression produce well-calibrated models?

Question

I understand that one of the reason logistic regression is frequently used for predicting click-through-rates on the web is that it produces well-calibrated models. Is there a good mathematical explanation for this?

Answer 1

Yes.

The predicted probability vector $p$ from logistic regression satisfies the matrix equation

$$ X^t(p - y) = 0$$

Where $X$ is the design matrix and $y$ is the response vector. This can be viewed as a collection of linear equations, one arising from each column of the design matrix $X$.

Specializing to the intercept column (which is a row in the transposed matrix), the associated linear equation is

$$ \sum_i( p_i - y_i) = 0 $$

so the overall average predicted probability is equal to the average of the response.

More generally, for a binary feature column $x_{ij}$, the associated linear equation is

$$ \sum_i x_{ij}(p_i - y_i) = \sum_{i \mid x_{ij} = 1}(p_i - y_i) = 0$$

so the sum (and hence average) of the predicted probabilities equals the sum of the response, even when specializing to those records for which $x_{ij} = 1$.

Answer 2

I think I can provide you an easy-to-understand explanation as follows:

We know that its loss function can be expressed as the following function:
$$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[ y^{(i)}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\log\left(1-h_\theta \left(x^{(i)}\right)\right)\right] $$
Where m represents the number of all the training cases, $y^{(i)}$ the label of the ith case, $h_{\theta}(x^{(i)})$ the predicted probability of the ith case: $\frac{1}{1+\exp[-\alpha -\sum_j \theta_j x^{(i)}_j]}$. (notice the bias $\alpha$ here)

Since the goal of training is to minimize the loss function, let us evaluate its partial derivative with respect to each parameter $\theta_j$(the detailed derivation can be found here):
$$\frac{\partial J(\theta)}{\partial \theta_j}=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)}$$
And setting it to zero yeils:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_j^{(i)}=\sum_{i=1}^m y^{(i)}\,x_j^{(i)}$$

That means that if the model is fully trained, the predicted probabilities we get for the training set spread themselves out so that for each feature the sum of the weighted (all) values of that feature is equal to the sum of the values of that feature of the positive observations.

The above fits every feature so as the bias $\alpha$. Setting $x_0$ as 1 and $\alpha$ as $\theta_0$ yeilds:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_0^{(i)}=\sum_{i=1}^m y^{(i)}\,x_0^{(i)}$$ Then we get: $$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)=\sum_{i=1}^m y^{(i)}$$ Where $h_\theta\left(x^{(i)}\right)$ is the predicted probability of the fully trained model for the ith case. And we can write the function in a compact way: $$\sum_{i=1}^m p^{(i)} =\sum_{i=1}^m y^{(i)}$$

We can see obviously that the logistic regression is well-calibrated.

Reference: Log-linear Models and Conditional Random Fields by Charles Elkan

Answer 3

Logistic regression models are typically trained by minimizing the negative log likelihood. Maximum-likelihood models are known to asymptotically identify the solution minimizing the Kullback-Leibler divergence $D_{KL}(P, Q)$ between the empirical data distribution $P$ and the distribution $Q$ described by the model, see wiki. Minimizing the KL divergence between the two distributions implies optimizing for calibration.