22
$\begingroup$

I understand that one of the reason logistic regression is frequently used for predicting click-through-rates on the web is that it produces well-calibrated models. Is there a good mathematical explanation for this?

$\endgroup$
3
  • 2
    $\begingroup$ logistic regression made to predict probabilities --> which lead to calibrated predictions if not overfit. while most machine learning models do not predict probabilites, but rather a classes - and there is some contortion to derived pseudo-probabilites from these predictions - > hence note well calibrated $\endgroup$
    – charles
    Apr 23 '16 at 5:10
  • 2
    $\begingroup$ I should have clarified in the question, but my question was more about why it's the case that LR is so useful for predicting probabilities. $\endgroup$ Apr 23 '16 at 16:19
  • 1
    $\begingroup$ It's worth noting that you can simply fit a logistic regression to the output of a poorly-calibrated classifier to get a calibrated model. This is called Platt Scaling en.wikipedia.org/wiki/Platt_scaling $\endgroup$ May 23 '19 at 12:19
18
$\begingroup$

Yes.

The predicted probability vector $p$ from logistic regression satisfies the matrix equation

$$ X^t(p - y) = 0$$

Where $X$ is the design matrix and $y$ is the response vector. This can be viewed as a collection of linear equations, one arising from each column of the design matrix $X$.

Specializing to the intercept column (which is a row in the transposed matrix), the associated linear equation is

$$ \sum_i( p_i - y_i) = 0 $$

so the overall average predicted probability is equal to the average of the response.

More generally, for a binary feature column $x_{ij}$, the associated linear equation is

$$ \sum_i x_{ij}(p_i - y_i) = \sum_{i \mid x_{ij} = 1}(p_i - y_i) = 0$$

so the sum (and hence average) of the predicted probabilities equals the sum of the response, even when specializing to those records for which $x_{ij} = 1$.

$\endgroup$
9
  • 2
    $\begingroup$ @MatthewDrury how can I interpret your first equation? is $p$ of the form $1/(1+\exp(-x))$?Nevertheless this linear relation holds? Thank you! $\endgroup$
    – Ric
    Feb 15 '18 at 11:16
  • 1
    $\begingroup$ Yes, p is of that form. The first equation comes from setting the derivative of the loss function to zero. $\endgroup$ Feb 15 '18 at 15:49
  • 2
    $\begingroup$ This addresses only calibration-in-the-large which is not what we want: calibration-in-the-small. $\endgroup$ Aug 20 '18 at 12:35
  • 1
    $\begingroup$ @FrankHarrell Care to elaborate? I haven't heard those terms before. $\endgroup$ Aug 20 '18 at 16:43
  • 6
    $\begingroup$ There's a long history in the probability forecast literature dating by to US Weather Service 1950 - that's where the Brier score was first use. Calibration-in-the-small means that if looked at predicted risks of 0.01, 0.02, ..., 0.99, each of these is accurate, i.e., for all the times when the predicted risk was 0.4, the outcome happened about 0.4 of the time. I call "calibration-in-the-tiny" the next step: for males in which the prediction was 0.4 was the outcome present 0.4 of the time, then for females. $\endgroup$ Aug 20 '18 at 22:22
8
$\begingroup$

I think I can provide you an easy-to-understand explanation as follows:

We know that its loss function can be expressed as the following function:
$$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[ y^{(i)}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\log\left(1-h_\theta \left(x^{(i)}\right)\right)\right] $$
Where m represents the number of all the training cases, $y^{(i)}$ the label of the ith case, $h_{\theta}(x^{(i)})$ the predicted probability of the ith case: $\frac{1}{1+\exp[-\alpha -\sum_j \theta_j x^{(i)}_j]}$. (notice the bias $\alpha$ here)

Since the goal of training is to minimize the loss function, let us evaluate its partial derivative with respect to each parameter $\theta_j$(the detailed derivation can be found here):
$$\frac{\partial J(\theta)}{\partial \theta_j}=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)}$$
And setting it to zero yeils:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_j^{(i)}=\sum_{i=1}^m y^{(i)}\,x_j^{(i)}$$

That means that if the model is fully trained, the predicted probabilities we get for the training set spread themselves out so that for each feature the sum of the weighted (all) values of that feature is equal to the sum of the values of that feature of the positive observations.

The above fits every feature so as the bias $\alpha$. Setting $x_0$ as 1 and $\alpha$ as $\theta_0$ yeilds:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_0^{(i)}=\sum_{i=1}^m y^{(i)}\,x_0^{(i)}$$ Then we get: $$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)=\sum_{i=1}^m y^{(i)}$$ Where $h_\theta\left(x^{(i)}\right)$ is the predicted probability of the fully trained model for the ith case. And we can write the function in a compact way: $$\sum_{i=1}^m p^{(i)} =\sum_{i=1}^m y^{(i)}$$

We can see obviously that the logistic regression is well-calibrated.

Reference: Log-linear Models and Conditional Random Fields by Charles Elkan

$\endgroup$
0
$\begingroup$

Logistic regression models are typically trained by minimizing the negative log likelihood. Maximum-likelihood models are known to asymptotically identify the solution minimizing the Kullback-Leibler divergence $D_{KL}(P, Q)$ between the empirical data distribution $P$ and the distribution $Q$ described by the model, see wiki. Minimizing the KL divergence between the two distributions implies optimizing for calibration.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.