3
$\begingroup$

I'm aware that if there is multicollinearity in the data with some correlated independent variables then there will be inaccuracies.

However, if I'm using the backward elimination model could I include all the highly correlated predictors and expect the best predictors to remain in the model at the end or the analysis or should I be removing one of the variables beforehand? If I should be removing a variable before carrying out backward elimination then how do I determine which variable(s) should be removed?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Yes, you can include all the correlated predictors. But as to my knowledge, there is no guarantee that "the best" predictors remain in the model. You just stepwise remove the predictor with the highes p-value (if higher than some threshold). At the end the selected variables might not optimise any reasonable criterion. So if you have some prior knowledge, you could use this knowledge to maybe throw out some variables beforehand.

Improve this answer
$\endgroup$
8
  • $\begingroup$ The issue is, I don't want to just throw out variables that may be of importance beforehand. For example, when I use backward elimination with all variables included, the first one I should remove with the highest p-value is a variable I think has some importance and is probably being affected by another variable. $\endgroup$
    – Ryan
    Commented Nov 30, 2016 at 13:14
  • 2
    $\begingroup$ How will you correct for the biases due to "fishing"? $\endgroup$
    – Frank Harrell
    Commented Nov 30, 2016 at 13:16
  • $\begingroup$ Then maybe backward selection is not the best way to go. It seems to me you have some prior knowledge (or at least an idea of which variables could be important). Why don't you just pick e.g. three models that make sense given your background knowledge and then compare those (e.g. using AIC)? @FrankHarrell If you know that two predictors essentially measure the same thing (and their correlation is higher than e.g. 0.9), I think it's ok to just include one of them in the model. $\endgroup$
    – Schlaftablette
    Commented Nov 30, 2016 at 13:34
  • $\begingroup$ @Schlaftablette No correlation between variables is higher than 0.8, although they are statistically significant. I have to use backward selection as part of my report (this was a requirement of the report that was set). Do you think it would therefore be acceptable to include all variables to begin with? $\endgroup$
    – Ryan
    Commented Nov 30, 2016 at 13:56
  • $\begingroup$ @Schlaftablette I'm not sure if it is noteworthy but one variable that appears to be correlated with a number of other variables is a composite variable I created (since I had 5 variables that were respective percentages of 5 "groups" and so I assigned a value from 1-5 for each group, multiplied that value by the percentage and summed them, then divided by 100 to create a single variable) $\endgroup$
    – Ryan
    Commented Nov 30, 2016 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.