2
$\begingroup$

I have a data set with highly positively correlated features that I'm classifying with LR. AFAIK correlated weights are not a problem in the same way they are in Naive Bayes - overcounting will not occur with LR.

The strange things that I'm seeing is that some of the highly correlated features assume opposite weights: feature A might be highly positive and feature B highly negative, though not as much. Is this a symptom of something going wrong with optimization or is this expected (a priori I expect A and B to be positive class indicators)

Improve this question
$\endgroup$
3
$\begingroup$

It is possible you are up against collinearity here (I'm assuming that when you say "correlated" you are assuming positive correlation, otherwise the postive/negative difference may make sense). In any case, caution should be used when confronting collinearity in logistic regression. Parameter estimates are often difficult to obtain and unreliable. Of course, this depends on how highly correlated your predictors are. To rule out collinearity, you might want to check something like the Variance Inflation Factor.

If your variables have a high correlation coefficient, but are not truly collinear, then it still isn't incredibly surprising to get the opposite sign behavior you observe (I say this without knowing more details of your problem), depending on what other variables are in your model. Remember that fitting an LR model fits all variables simultaneously to the outcome, so you typically have to interpret the weights as a whole. They may be correlated with each other, but have opposite effects in predicting an outcome, especially if grouped with other variables.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your response - yes I meant positively correlated, and I actually meant Logistic Regression (edited post to reflect both, my bad) $\endgroup$ – Maxim Khesin Oct 30 '13 at 17:54
  • $\begingroup$ With the change from linear to logistic regression, my comments above still hold (hold for any GLM). $\endgroup$ – ScatterSignalNoise Oct 30 '13 at 17:54
  • $\begingroup$ There is no matrix inversion in Logistic Regression training - I believe gradient methods are used for weight optimization. To give you a feel for the problem imagine using both bi- and tri- grams for "offensiveness" classification. "this sux" and "this sux s*t" would be highly positively correlated, both being useful features. $\endgroup$ – Maxim Khesin Oct 30 '13 at 17:59
  • $\begingroup$ @MaximKhesin Logistic regression can be trained with many methods, some of which involve matrix inversion, e.g. IRLS. (But that's kind of irrelevant to the problem here.) Anyway, it seems like logistic regression is deciding that "this sux", when it's not part of "this sux s*t" or similar, is actually a negative indicator. Are you sure that's not the case in your problem? You might also try running e.g. L2-regularized logistic regression, if you're not currently regularizing. $\endgroup$ – Danica Oct 30 '13 at 18:54
1
$\begingroup$

This might be a case of ceteris paribus confusion. It's hard to know without knowing more about your analysis.

Example 5 from Peter Kennedy might be relevant here:

In a linear regression of racehorse auction prices on various characteristics of the horse and information on its sire (father) and dam (mother), Robbins and Kennedy found that although the estimated coefficient on dam dollar winnings was positive, the coefficient on number of dam wins was negative, suggesting that yearlings from dams with more race wins are worth less. This wrong sign problem is resolved by recognizing that the negative sign means that holding dam dollar winnings constant, a yearling is worth less if its dam required more wins to earn those dollars.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.