1
$\begingroup$

This is a pretty straightforward question and I guess I will get a negative score here - I was so happy improving my points in here lol -, but I couldn't find anywhere and even though I believe I know the answer, I would like to assess with more senior Data Scientists.

When dealing with Linear Regression, it's often recommended to remove features that presents multicollinearity so we could get the correct interpretability (even though it's not a bias problem). The way we do this is using Variance Inflation Factor. Should we do this recursively? I mean, should I check VIF, then remove some of then, run again, remove others and so on? Is there anything we need to check besides VIF in this process?

$\endgroup$

1 Answer 1

3
$\begingroup$

You can remove them recursively and I think it is advisable to do so. I don’t have a paper to support this idea but you can consider the following rationale. VIF check if one input is correlated with at least another output. It doesn’t tell you, however, with which other inputs is correlated. You might have a single group of correlated variables or more. Also after you remove one input you have to recompute VIF for all remaining inputs because the input you removed might be involved. The only reasonable way I found is to do multiple iterations. At each iteration compute VIF for each feature. See if there is at least one with coefficient value considered as candidate for removal. If there is none you stop. If there are more, than you chose the one with the extreme value and iterate again. For me it produced more than reasonable results.

$\endgroup$
2
  • 1
    $\begingroup$ Concerning the last sentence: Do you have a sample dataset on which this works well and which can be used to demonstrate the method? $\endgroup$
    – cdalitz
    Aug 31, 2021 at 7:41
  • $\begingroup$ No, but you can simulate various multiple colinearity situations. Start with some independent inputs, make some dependent ones, add noise and try the procedure. $\endgroup$
    – rapaio
    Sep 1, 2021 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.