What to do when every VIF value is infinity

Question

I know that VIF values have no upper limit, and that anything over 10 is usually bad news if you are trying to avoid multicollinearity especially for regression models such as multiple logistic regression.

However - most of what I have read suggests removing the offending infinity variable. Here in lies the problem.

I'm currently working on a dataset with nearly 2000 variables, and every single one has produced a VIF of infinity.

I've been doing this on python:

vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(df.values, i) for i in range(dif.shape[1])]
vif_info['Column'] = df.columns
vif_info.sort_values('VIF', ascending=False)

and I have tried various different methods, which have all produced the same results, so I'm relatively sure I haven't done something wrong there.

I have also tried log transforming the data and I still get the same result.

I'm not sure if it adds up though because when I produce something like a clustermap using a regular correlation matrix (see below) there seems to be clear specific variables that are highly correlated but not all variables seem to be this way.

In terms of statistical analysis I'm not sure how one would proceed with these results (or if theres something obvious that I've done wrong).

Answer 1

It seems that a somevariables are able to create perfect multiple regressions on other variables (which would explain why all the VIF are infinity).

In order to identify them, I would try to do some actual regressions $X_j=X_{\backslash j}\beta+\epsilon$ and check the coefficients in order to try to identify the problematic variables.

Otherwise, maybe this comes from your dataset. I am not familiar with chemistry data at all. According to your experience, is it common to have very high VIF in such a dataset ? Also, what is the size of your dataset ? How big is the number of observations in comparison to your 2000 regressors ?

EDIT:

Based on your comment, it is likely that the issue comes from the fact that you have way more variables than observations ($k=2000 > N = 45$). So all the regressions end up having $R^2=1$, which corresponds to your VIF equal to 1 for all variables.

As a consequence, I suggest you try to find a way to use a small number of regressors for your regressions. A technique like forward stepwise regression could help, but will prevent you from doing inference with your resulting model. This will be fine if you are only interested in predictions.

Answer 2

If you were going for prediction, I would have suggested some regularization approach.

In a context of Null Hypothesis Significance Testing (NHST), you may want to look at the collinearity diagnostics of Belsley, Kuh and Welsch. In contrast to VIF, they also reveal which groups of predictors are closely associated. This makes deciding how to "thin out" your predictor set much more focused. The disadvantage is that BKW is quite a bit more complicated. And I don't know how well it works with 2000 predictors.

Alternatively, you could use a dimension reduction technique like PCA and run your logistic regression on the first few principal components. The disadvantage here is that you get the p values of the principal components, not of your original data.

Finally, NHST really presupposes you have predefined hypotheses in the context of a planned experiment. If you really want to feed in 2000 predictors and assess whether any coefficient is nonzero, perhaps you should go back to the planning stage and plan your experimental controls more carefully.