13
$\begingroup$

I understand that one of the reason logistic regression is frequently used for predicting click-through-rates on the web is that it produces well-calibrated models. Is there a good mathematical explanation for this?

$\endgroup$
  • 2
    $\begingroup$ logistic regression made to predict probabilities --> which lead to calibrated predictions if not overfit. while most machine learning models do not predict probabilites, but rather a classes - and there is some contortion to derived pseudo-probabilites from these predictions - > hence note well calibrated $\endgroup$ – charles Apr 23 '16 at 5:10
  • 2
    $\begingroup$ I should have clarified in the question, but my question was more about why it's the case that LR is so useful for predicting probabilities. $\endgroup$ – lsankar4033 Apr 23 '16 at 16:19
  • $\begingroup$ It's worth noting that you can simply fit a logistic regression to the output of a poorly-calibrated classifier to get a calibrated model. This is called Platt Scaling en.wikipedia.org/wiki/Platt_scaling $\endgroup$ – generic_user May 23 '19 at 12:19
14
$\begingroup$

Yes.

The predicted probability vector $p$ from logistic regression satisfies the matrix equation

$$ X^t(p - y) = 0$$

Where $X$ is the design matrix and $y$ is the response vector. This can be viewed as a collection of linear equations, one arising from each column of the design matrix $X$.

Specializing to the intercept column (which is a row in the transposed matrix), the associated linear equation is

$$ \sum_i( p_i - y_i) = 0 $$

so the overall average predicted probability is equal to the average of the response.

More generally, for a binary feature column $x_{ij}$, the associated linear equation is

$$ \sum_i x_{ij}(p_i - y_i) = \sum_{i \mid x_{ij} = 1}(p_i - y_i) = 0$$

so the sum (and hence average) of the predicted probabilities equals the sum of the response, even when specializing to those records for which $x_{ij} = 1$.

$\endgroup$
  • 1
    $\begingroup$ @MatthewDrury how can I interpret your first equation? is $p$ of the form $1/(1+\exp(-x))$?Nevertheless this linear relation holds? Thank you! $\endgroup$ – Ric Feb 15 '18 at 11:16
  • 1
    $\begingroup$ Yes, p is of that form. The first equation comes from setting the derivative of the loss function to zero. $\endgroup$ – Matthew Drury Feb 15 '18 at 15:49
  • 1
    $\begingroup$ This addresses only calibration-in-the-large which is not what we want: calibration-in-the-small. $\endgroup$ – Frank Harrell Aug 20 '18 at 12:35
  • 1
    $\begingroup$ @FrankHarrell Care to elaborate? I haven't heard those terms before. $\endgroup$ – Matthew Drury Aug 20 '18 at 16:43
  • 3
    $\begingroup$ There's a long history in the probability forecast literature dating by to US Weather Service 1950 - that's where the Brier score was first use. Calibration-in-the-small means that if looked at predicted risks of 0.01, 0.02, ..., 0.99, each of these is accurate, i.e., for all the times when the predicted risk was 0.4, the outcome happened about 0.4 of the time. I call "calibration-in-the-tiny" the next step: for males in which the prediction was 0.4 was the outcome present 0.4 of the time, then for females. $\endgroup$ – Frank Harrell Aug 20 '18 at 22:22
2
$\begingroup$

I think I can provide you an easy-to-understand explanation as follows:

We know that its loss function can be expressed as the following function:
$$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[ y^{(i)}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\log\left(1-h_\theta \left(x^{(i)}\right)\right)\right] $$
Where m represents the number of all the training samples, $y^{(i)}$ the label of the ith sample, $h_{\theta}(x^{(i)})$ the predicted probability of the ith sample: $\frac{1}{1+\exp[-\alpha -\sum_j \theta_j x^{(i)}_j]}$. (notice the bias $\alpha$ here)

Since the goal of training is to minimize the loss function, let us evaluate its partial derivative with respect to each parameter $\theta_j$(the detailed derivation can be found here):
$$\frac{\partial J(\theta)}{\partial \theta_j}=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)}$$
And setting it to zero yeils:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_j^{(i)}=\sum_{i=1}^m y^{(i)}\,x_j^{(i)}$$

That means that if the model is fully trained, the predicted probabilities we get for the training set spread themselves out so that for each feature the sum of the weighted (all) values of that feature is equal to the sum of the values of that feature of the positive samples.

The above fits every feature so as the bias $\alpha$. Setting $x_0$ as 1 and $\alpha$ as $\theta_0$ yeilds:
$$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)x_0^{(i)}=\sum_{i=1}^m y^{(i)}\,x_0^{(i)}$$ Then we get: $$\sum_{i=1}^m h_\theta\left(x^{(i)}\right)=\sum_{i=1}^m y^{(i)}$$ Where $h_\theta\left(x^{(i)}\right)$ is the predicted probability of the fully trained model for the ith sample. And we can write the function in a compact way: $$\sum_{i=1}^m p^{(i)} =\sum_{i=1}^m y^{(i)}$$

We can see obviously that the logistic regression is well-calibrated.

Reference: Log-linear Models and Conditional Random Fields by Charles Elkan

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.