0
$\begingroup$

I'm working with an essentially linear unsupervised modeling approach which (predictably) has problems when there is (multi)-collinearity. To avoid this, I've written some code which removes variables one-by-one based on which reduces VIF the most. I.e.

The function VIF(X, i) returns the VIF of the column i of X when regressed against the other columns. Also, assume we have.

max_VIF = max([VIF(X, i) for i in columns of X])

while max_VIF > threshold:
    best_VIF = infinity
    remove_col = None
    for i in columns of X:
       Y = X without column i
       Y_VIF = max([VIF(Y, i) for i in columns of Y])
       if Y_VIF < best_VIF:
           best_VIF = Y_VIF
           remove_col = i
    remove i from X
    max_VIF = best_VIF

This, unfortunately, requires me to calculate the VIF several times which is a VERY slow process. Is there a better way to do this?

Edit: making pseudo code more clear.

Improve this question
$\endgroup$
10
  • $\begingroup$ Linear unsupervised model? Which model is this? What is your goal, apart from fitting a model successfully? Why are you doing this? $\endgroup$
    – user2974951
    Commented Feb 9, 2023 at 16:42
  • $\begingroup$ It is not clear that you should do this at all. What problems are there when you have multicollinearity, and how does removing variables remedy those issues? Consider reading my CW post here, the latter half of which addresses feature dependence and links to additional material (which also contains further reading). $\endgroup$
    – Dave
    Commented Feb 9, 2023 at 16:57
  • $\begingroup$ @user2974951 the model is a causal discovery algorithm which assumes linearity. This specific model isn't one that is very well known and I only have access to it as a black box. However, I think the same issue would arise with the PC algorithm if the implemented with linear regression to check for independence. My overall goal, at least at the moment, is to "find something interesting in the resulting causal graph." $\endgroup$
    – roundsquare
    Commented Feb 9, 2023 at 17:37
  • $\begingroup$ @Dave in general, I agree. However, LASSO and and ridge regression both require a target variable (at least as far as I know). When I have one, I do use these techniques pretty frequently, but right now I don't have one (this is all somewhat preliminary). I'm open to other techniques as well. $\endgroup$
    – roundsquare
    Commented Feb 9, 2023 at 17:39
  • 2
    $\begingroup$ But why care about the classical VIF? In linear models, multicollinearity has drawbacks, but removing features introduces issues that might make you worse off for having removed the features. However, at least VIF has an interpretation there, in that it literally refers to the factor by which coefficient variances are inflated compared to when the features are independent. When you are not in a situation where linear modeling is occurring, the interest in VIF does not necessarily make sense (despite the calculation of VIF only involving the features and not the outcome, if there even is one). $\endgroup$
    – Dave
    Commented Feb 9, 2023 at 21:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.