Do coefficients of logistic regression have a meaning?

Question

I have a binary classification problem from several features. Do the coefficients of a (regularized) logistic regression have an interpretable meaning?

I thought they could indicate the size of influence, given the features are normalized beforehand. However, in my problem the coefficients seem to depend sensitively on the features I select. Even the sign of the coefficients changes with different feature sets chosen as input.

Does it make sense to examine the value of the coefficients and what is the correct way to find the most meaningful coefficients and state their meaning in words? Are some fitted models and their sign of the coefficients wrong - even if when they sort-of fit the data?

(The highest correlation that I have between features is only 0.25, but that certainly plays a role?)

Answer 1

The coefficients from the output do have a meaning, although it isn't very intuitive to most people and certainly not to me. That is why people change them to odds ratios. However, the log of the odds ratio is the coefficient; equivalently, the exponentiated coefficients are the odds ratios.

The coefficients are most useful for plugging into formulas that give predicted probabilities of being in each level of the dependent variable.

e.g. in R

library("MASS")
data(menarche)
glm.out = glm(cbind(Menarche, Total-Menarche) ~ Age,
                family=binomial(logit), data=menarche)

summary(glm.out)

The parameter estimate for age is 1.64. What does this mean? Well, if you combine it with the parameter estimate for the intercept (-21.24) you can get a formula predicting the likelihood of menarche:

$P(M) = \frac{1}{1 + e^{21.24 - 1.64*age}}$

but that formula (even with just one variable!) doesn't give much of a sense of how age is related to menarche. If we use the odds ratio (which is $e^{1.64} = 5.16$ that means that, for each additional year of age, the odds of menarche are 5.16 times as big (not exactly 5.16 times as likely, but that interpretation is often used).

Answer 2

Interpreting directly the coefficients is difficult and can be misleading. You have no guarantees on how weights are assigned among the variables.

Quick example, similar to the situation you describe: I have worked on a model of the interaction of users to a website. That model included two variables that represent the number of "clicks" during the first hour and during the second hour of a user session. These variables are highly correlated to each other. If both coefficients for those variable were positive then we could easily mislead ourselves and believe that maybe higher coefficient indicates "higher" importance. However, by a adding/removing other variables we could easily end up with a model where the first variable had positive sign and the other negative. The reasoning we ended up to was that since there were some significant (albeit low) correlations between most pairs of the available variables we couldn't have any secure conclusion on the importance of the variables using the coefficients (happy to learn from the community if this interpretation is correct).

If you want to get a model where it is kind of easier to interpret one idea would be to use Lasso (minimisation of the L1 norm). That leads to sparse solutions were variables are less correlated to each other. However, that approach wouldn't easily pick both variables of the previous example - one would be zero wighted.

If you just want to assess the importance of specific variables, or sets of variables, I would recommend using directly some feature selection approach. Such approaches lead to much more meaningful insights and even global rankings of the importance of the variables based on some criterion.

Answer 3

The coefficients most certainly have a meaning. In some software packages the model can be directed in either of two ways to produce either of two types of coefficients. For example, in Stata, one can use either the Logistic command or the logit command; in using one, the model gives traditional coefficients, while in using the other, the model gives odds ratios.

You may find that one is much more meaningful to you than the other.

About your question that "...coefficients seem to depend sensitivity...".

Are you saying that the results depend on what variables you put in the model?

If so, yes, this is a fact of life when doing regression analysis. The reason for this is that regression analysis is looking at a bunch of numbers and crunching them in an automated way.

The results depend on how the variables are related to each other and on what variables are not measured. It is as much an art as it is a science.

Furthermore, if the model has too many predictors compared to the sample size, the signs can flip around in a crazy way - I think of this is saying that the model is using variables that have a small effect to "adjust" its estimates of those that have a big effect (like a small volume knob to make small calibrations). When this happens, I tend to not trust the variables with small effects.

On the other hand, it may be that signs initially change, when you add new predictors, because you are getting closer to the causal truth.

For example, lets imagine that Greenland Brandy might be bad for one's health but income is good for one's health. If income is omitted, and more rich people drink Brandy, then the model may "pick up" the omitted income influence and "say" that the alcohol is good for your health.

Have no doubt about it, it is a fact of life that coefficients depend on the other variables that are included. To learn more, look into "omitted variable bias" and "spurious relationship". If you have not encountered these ideas before, try to find introduction to statistics courses that meet your needs - this can make a huge difference in doing the models.