1
$\begingroup$

This might be noob question, because I've just started to learn data analytics.

Why should we or should we not include a correlated predictor in our model?

While selecting predictors, for a class project, I observed that 4 variables exhibited more than 90% correlation with each other. I was using logistic regression for my model. And I only included one variable which gave me the lowest misclassification error during the kfold cross validation.

However, my classmates who included one other correlated variables as well the other predictors which I used, performed better on the Test Data.

On reflection, I reasoned that maybe the response could be function of both the correlated pairs ( something like y = ax + bx^2 + c).

I am still not wholly convinced of it. Could someone shed more light on this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

This is a 'for what it's worth' answer. Firstly, I believe one should be cautious with highly correlated variables and logistic regression. I've heard, and have my own anecdotal evidence, of logistic regression not doing a very good job with correlated or too many variables. Having said that, I think you have answered your own question. You have variables, build all the models and let either cross validation or, better yet, a train/test analysis of the data, decide on which model is best. I know one can also bring AIC into the picture for logistic regression in R.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Every technique will do badly with correlated and/or too many variables if it is not regularized. Logistic regression can be regularized. Moreover, when data is abundant and collinearity is low, regularization is less effective, with the extreme that when every single data point is known (no unobserved cases exist), a nonregularized model will make the best predictions. $\endgroup$
    – jona
    Commented Dec 12, 2015 at 22:43
  • $\begingroup$ @ jona I agree with everything you say, but different techniques do better than others. I certainly don't know a mathematical proof, but I believe that, for example, random forests should specifically help with correlated variables as should neural nets. My main point was about the use of CV or train/test to answer the question. $\endgroup$
    – meh
    Commented Dec 16, 2015 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.