Does Regularized Logistic Regression Produce Calibrated Results? It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow when there is a regularization involved. Can somebody please tell me if indeed it is the case that a regularized logistic regression model is not calibrated?
 A: The answer depends entirely on the amount of penalization used.  If too little, the model will be seen to be overfitted when evaluated on an independent sample.  If too much penality, it will be found to be underfitted.  The goal is to solve for the penalty that gets it "just right".  Two ways to do this are cross-validation (e.g., 100 repeats of 10-fold cross-validation) or computing the effective AIC and solving for the penalty that optimizes it.
The calibration curve is the plot of x=predicted probability of the event vs. y=actual probability of the event.  The actual probability is obtained (in an independent sample, or using the bootstrap to correct the "apparent" calibration) by running a smoother on $(\hat{P}, Y)$, where $Y$ is the vector of binary outcomes.  If the calibration curve is linear, it can be summarized by its intercept and slope.  When the slope is greater than 1 there is underfitting, and when the slope is less than 1 there is overfitting (regression to the mean; low $\hat{P}$ are too low and high ones are too high).
The Bayesian approach to this has major advantages over what is described above, including:


*

*the penalty is automatically estimated if you do a reasonable job of specifying a prior distribution for the penalty parameters (usually stated in terms of the reciprocal, i.e., the variance of random effects)

*once finished, the Bayesian posterior distribution works exactly as it should, whereas the frequentist approach does not give us confidence intervals or hypothesis tests once penalization is used

