I understand the coefficients of a logistic equation can be interpreted as odd ratio. If a regularization term is added to control for over-fitting, how does this change the interpretation of the coefficients?


The coefficients that are returned standard with a logistic regression fit are not odds ratios. They represent the change in the log odds of 'success' associated with a one-unit change in their respective variable, when all else is held equal. If you exponentiate a coefficient, then you can interpret the result as an odds ratio (of course, this is not true of the intercept). More on this can be found in my answer here: Interpretation of simple predictions to odds ratios in logistic regression.

Adding a penalty to the model fit will (potentially) change the fitted value of the estimated coefficients, but it does not change the interpretation of the coefficients in the sense discussed in your question / above.*

* (I wonder if confusion about this statement is the origin of the recent downvote.) To be clearer: The fitted coefficient on $X_1$, $\hat\beta_1$ represents the change in the log odds of success associated with a 1-unit change in $X_1$ if there is no penalty term used in fitting the model and if a penalty term is used to fit the model. In neither case is it the odds ratio. However, $\exp(\hat\beta_1)$ is the odds ratio associated with a 1-unit change in $X_1$, again irrespective of whether a penalty term was used to fit the model. A model fitted with a penalty term can be interpreted within a Bayesian framework, but doesn't necessarily have to be. Moreover, even if it is, $\hat\beta_1$ still represents the change in the log odds of success associated with a 1-unit change in $X_1$ not an odds ratio.

  • 1
    $\begingroup$ I never understood why people downvote without leaving a comment. Anyway, this is a great answer. $\endgroup$ – Digio Apr 6 '18 at 21:33

Regularized linear regression and regularized logistic regression can be interpreted nicely from a Bayesian point of view. The regularization parameter corresponds to a choice of prior distribution on the weights, for example, something like a normal distribution centered at zero with standard deviation given by the inverse of the regularization parameter. Then via your training data, these distributions are updated to finally give you the posterior distributions on the weights.

So, for example, a larger regularization parameter means that, as a prior, we think that the weights should be closer to zero, hence with this setup it's less likely that the posterior distributions will be supported far away from zero -- which agrees with the intuition of what regularization is "supposed to do".

For most implementations of regularized regression, the final output of the weights is just the expected value of the posterior distributions.

By the way, unregularized regression can be basically interpreted in the same way: it's the limit as the regularization parameter goes to zero.

  • $\begingroup$ That is a nice way to interpret regularized linear estimates. Does your answer change at all when the "prior" (regularization parameter) is selected based on the data (eg, through cross validation)? $\endgroup$ – Andrew M Dec 9 '14 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.