2
$\begingroup$

I learned that regularized logistic regression helps prevent the model from over-fitting the data. I understand that the function is still technically a high-order polynomial, but the effect is reduced so it looks more like a curve, but here's the part where I may understand incorrectly: The function is regularized by adding a term to the end that penalizes the rest of the parameters.

In a sense, the way I think of this is imagining a sine wave with a very small coefficient, like 0.0001, so the graph of 0.00001 * sin(x) would look much like a straight line compared to the same perspective of sin(x).

Is this the correct way to look at how regularized logistic regression works, or does the regularization follow some other principle?

$\endgroup$
5
$\begingroup$

You are mixing various ideas. Logistic regression uses the logistic function to translate the unrestricted linear predictor to a restricted $[0,1]$ space. This function must be monotonic. The $X$s on the other hand can include complex functions.

Regularization is better called penalized maximum likelihood estimation because its purpose nowadays is to avoid overfitting and no so much to regularize (make variables less correlated). You can use penalized MLE with logistic models even if all the $X$s affect the linear predictor (log odds that $Y=1$) linearly. Note that penalization = shrinkage.

If you have multiple terms representing one predictor (say using a regression spline), penalization will make the resulting fitted transformation more flat. Targeted penalization (e.g., penalizing only the nonlinear components) will make it more simple but still possibly steep.

$\endgroup$
2
  • 1
    $\begingroup$ (+1) I don't understand the sentence "this function must be monotonic." Does that mean that the logistic function is monotonic, or are you asserting that there is some mathematical or statistical requirement that any function used to "translate the unrestricted linear predictor to a restricted $[0,1]$ space" has to be monotonic? If it's the latter, when why is my answer at stats.stackexchange.com/a/64039 not a counterexample? $\endgroup$ – whuber Feb 6 '16 at 16:21
  • 1
    $\begingroup$ I guess you could have a non-monotonic one but then predictors would often have to be transformed non-monotonically. But there is seldom need for a non-monotonic function to transform $X\beta$ to $P$. The sunflower example, to me, is an excellent example of why I like regression splines to relax the linearity assumption (and monotonicity assumption) in $X$. $\endgroup$ – Frank Harrell Feb 6 '16 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.