Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a binary classification problem from several features. Do the coefficients of a (regularized) logistic regression have an interpretable meaning?

I thought they could indicate the size of influence, given the features are normalized beforehand. However, in my problem the coefficients seem to depend sensitively on the features I select. Even the sign of the coefficients changes with different feature sets chosen as input.

Does it make sense to examine the value of the coefficients and what is the correct way to find the most meaningful coefficients and state their meaning in words? Are some fitted models and their sign of the coefficients wrong - even if when they sort-of fit the data?

(The highest correlation that I have between features is only 0.25, but that certainly plays a role?)

share|improve this question
    
Could you clarify what you mean by regularised? Do you have an L2 penalty term, and if so have you searched for the optimal factor eg by cross validation? –  seanv507 Jun 28 at 22:46
    
Yes, I allow for L2 penalty terms on the coefficients. I did search for the optimal regularization factor, but I haven't used feature selection (like forward selection) yet. However, it makes me feel uncertain about it, since the coefficients depend so sensitively on the choice of features I include. Assuming that each feature has either positive or negative effect of the positive class, how can I determine their strength and direction? –  Gerenuk Jul 1 at 14:02
add comment

2 Answers 2

The coefficients from the output do have a meaning, although it isn't very intuitive to most people and certainly not to me. That is why people change them to odds ratios. However, the log of the odds ratio is the coefficient; equivalently, the exponentiated coefficients are the odds ratios.

The coefficients are most useful for plugging into formulas that give predicted probabilities of being in each level of the dependent variable.

e.g. in R

library("MASS")
data(menarche)
glm.out = glm(cbind(Menarche, Total-Menarche) ~ Age,
                family=binomial(logit), data=menarche)

summary(glm.out)

The parameter estimate for age is 1.64. What does this mean? Well, if you combine it with the parameter estimate for the intercept (-21.24) you can get a formula predicting the likelihood of menarche:

$P(M) = \frac{1}{1 + e^{21.24 - 1.64*age}}$

but that formula (even with just one variable!) doesn't give much of a sense of how age is related to menarche. If we use the odds ratio (which is $e^{1.64} = 5.16$ that means that, for each additional year of age, the odds of menarche are 5.16 times as big (not exactly 5.16 times as likely, but that interpretation is often used).

share|improve this answer
    
I will edit my answer –  Peter Flom Jun 28 at 22:03
add comment

Interpreting directly the coefficients is difficult and can be misleading. You have no guarantees on how weights are assigned among the variables.

Quick example, similar to the situation you describe: I have worked on a model of the interaction of users to a website. That model included two variables that represent the number of "clicks" during the first hour and during the second hour of a user session. These variables are highly correlated to each other. If both coefficients for those variable were positive then we could easily mislead ourselves and believe that maybe higher coefficient indicates "higher" importance. However, by a adding/removing other variables we could easily end up with a model where the first variable had positive sign and the other negative. The reasoning we ended up to was that since there were some significant (albeit low) correlations between most pairs of the available variables we couldn't have any secure conclusion on the importance of the variables using the coefficients (happy to learn from the community if this interpretation is correct).

If you want to get a model where it is kind of easier to interpret one idea would be to use Lasso (minimisation of the L1 norm). That leads to sparse solutions were variables are less correlated to each other. However, that approach wouldn't easily pick both variables of the previous example - one would be zero wighted.

If you just want to assess the importance of specific variables, or sets of variables, I would recommend using directly some feature selection approach. Such approaches lead to much more meaningful insights and even global rankings of the importance of the variables based on some criterion.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.