It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow when there is a regularization involved. Can somebody please tell me if indeed it is the case that a regularized logistic regression model is not calibrated?
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ This answer is perhaps more relevant: stats.stackexchange.com/a/16541. In a lot of regularised logistic regression I have seen (e.g. glmnet), the regularisation is not applied to the intercept. So, I do not see why the calibration should be affected in the case of regularisation. We should be able to check this by repeating cardinal's calculation using the penalised log-likelihood instead of the log-likelihood. $\endgroup$– AlexCommented Aug 23, 2018 at 4:57
-
1$\begingroup$ @Alex, thanks for your input. Cardinl's reasoning in my humble opinion does not follows since after taking derivates there will be an extra term of $+\lambda \beta_i$ in the equations instead of zero on the other side. In other words the regularization factor of. $\lambda \|\beta\|^2$ will not cancel when taking derivatives. $\endgroup$– CupitorCommented Aug 24, 2018 at 0:02
-
2$\begingroup$ Maybe I am being dense here, but I don't see why, for the constant term, i.e. $\beta_0$, there would be an "extra term of $+\lambda\beta_0$"? The intercept is not usually included in the penalty term so shouldn't the penalty terms be treated as a constant with respect to the partial derivative? $\endgroup$– AlexCommented Aug 24, 2018 at 1:39
-
1$\begingroup$ See for example, stats.stackexchange.com/a/161689, which deals with the linear regression case. $\endgroup$– AlexCommented Aug 24, 2018 at 1:40
-
$\begingroup$ Oh I am totally sorry. I get it now. Fair enough! That solves the issue :) $\endgroup$– CupitorCommented Aug 24, 2018 at 7:27
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
11
The answer depends entirely on the amount of penalization used. If too little, the model will be seen to be overfitted when evaluated on an independent sample. If too much penality, it will be found to be underfitted. The goal is to solve for the penalty that gets it "just right". Two ways to do this are cross-validation (e.g., 100 repeats of 10-fold cross-validation) or computing the effective AIC and solving for the penalty that optimizes it.
The calibration curve is the plot of x=predicted probability of the event vs. y=actual probability of the event. The actual probability is obtained (in an independent sample, or using the bootstrap to correct the "apparent" calibration) by running a smoother on $(\hat{P}, Y)$, where $Y$ is the vector of binary outcomes. If the calibration curve is linear, it can be summarized by its intercept and slope. When the slope is greater than 1 there is underfitting, and when the slope is less than 1 there is overfitting (regression to the mean; low $\hat{P}$ are too low and high ones are too high).
The Bayesian approach to this has major advantages over what is described above, including:
- the penalty is automatically estimated if you do a reasonable job of specifying a prior distribution for the penalty parameters (usually stated in terms of the reciprocal, i.e., the variance of random effects)
- once finished, the Bayesian posterior distribution works exactly as it should, whereas the frequentist approach does not give us confidence intervals or hypothesis tests once penalization is used
answered Aug 20, 2018 at 12:40
-
1$\begingroup$ Thank you. But I am not sure if I understand the connection of your answer to calibration. No matter how the regularization coefficient is chosen eventually we end up with a learned model based on optimization. The question is whether that model is calibrated meaning that $\hat{p}(y=1)\approx p(y=1)$. $\endgroup$– CupitorCommented Aug 20, 2018 at 22:39
-
1$\begingroup$ Me too. I don't see how that answer relates to calibration (maybe it's implicit in the answer). I know, empirically, that constrained regressions (e.g. using an offset term) are not necessarily calibrated, and I just performed a penalised regression that is close to, but not exactly, calibrated on the training sample (let alone a hold-out sample). $\endgroup$ Commented Aug 21, 2018 at 0:15
-
$\begingroup$ @RossGayler thank you for sharing the sentiment. I was confused. I can see that the proof does not follow so it is not surprising. But it is surprising that there is nothing written about it out there. $\endgroup$– CupitorCommented Aug 21, 2018 at 20:11
-
$\begingroup$ In light of these comments, Frank, could you update this answer to make any implicit connections to calibration easier for readers to perceive? $\endgroup$ Commented Aug 21, 2018 at 20:27
-
1$\begingroup$ I agree with @frank-harrell. Perhaps you are working from a different definition of calibration. Your (Cupitor) definition in the first comment after Frank's answer only requires that the global mean predicted probability be correct. Whereas the definition that Frank (and I) use is that the actual probability conditional on the predicted probability should equal the predicted probability. Hence Frank's reference to "the calibration curve". Anyway, empirically I have in front of me right now two calibration curves for the same regression: unpenalised (slope=1), penalised (slope=1.06). $\endgroup$ Commented Aug 24, 2018 at 22:18