3
$\begingroup$

In my dataset, all the variables are highly correlated (correlation coefficient > 0.95). However, the correlation with the dependent variables is very low (<0.35). I checked the variation inflation factor (VIF), but it's 'inf' for some variables and very high (around 6 figures) for other variables. So cannot filter with the threshold of 10 as usual but setting a threshold around 1000 does not seem to be correct either. The response is a binary outcome. I'm trying to build a logistic regression model but due to high collinearity coefficients are not interpretable. How can I handle multi-collinearity in this case?

EDIT: I want to use the model to check what variables are significantly associated with the outcome and also to predict the outcome. The training set size is 128 (after splitting into train-test) and there are 96 variables in total. Data is highly imbalanced.

Improve this question
$\endgroup$
6
  • $\begingroup$ What is your goal with the model, what will you use it for? $\endgroup$
    – user2974951
    Commented Mar 1, 2023 at 15:46
  • $\begingroup$ I want to use it to check what risk factors are significantly associated with the outcome and also for the prediction of the outcome. $\endgroup$
    – Dushi Fdz
    Commented Mar 1, 2023 at 15:50
  • $\begingroup$ How are "risk factors" related to the "variables" you refer to in this question? Are they synonyms, or are risk factors perhaps functions of one or more variables? $\endgroup$
    – whuber ♦
    Commented Mar 1, 2023 at 15:59
  • 1
    $\begingroup$ It would be helpful to see the singular values of the centered design matrix (aka the PCA "scree plot"). This will help us understand "how many" variables there "actually". $\endgroup$
    – John Madden
    Commented Mar 1, 2023 at 16:06
  • $\begingroup$ How many predictor variables are involved? How many cases? $\endgroup$
    – EdM
    Commented Mar 1, 2023 at 16:25

1 Answer 1

Reset to default
8
$\begingroup$

The correlations and VIFs are telling you the story. All of your features are extremely related to each other, so any notion of increasing one while holding others constant is nonsense, and I see no hope for untangling your features.

It’s nice to be able to have a clean interpretation, but the universe does not owe you one.

Further, you have way too many variables for your sample size. Even if you combined your training and testing data, you would have more variables than instances of the minority class. I fear that whoever tasked you with doing this has, probably unknowingly, sent you on a snipe hunt.

Finally, while VIF gives some idea of how features are related, in a logistic regression, the usual VIF does not have its literal interpretation as the factor by which the coefficient variance is inflated. When you work on a more reasonable logistic regression problem, you might want to consider generalizations of the usual VIF.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ Nicely said. (+1) This page provides links related to the generalized VIF appropriate for logistic regression. The data here are too few for a train/test split, just as you indicate. Substantial data reduction or penalization (ridge, LASSO) are called for. $\endgroup$
    – EdM
    Commented Mar 1, 2023 at 21:09
  • $\begingroup$ Thank you so much! In the case of high multi-collinearity, is there a preferred method for feature selection if I use machine learning models for the prediction of the binary outcome? Can I still use methods like Recursive Feature Elimination with cross-validation (RFE-CV)? $\endgroup$
    – Dushi Fdz
    Commented Mar 2, 2023 at 15:47
  • $\begingroup$ Features selection, in general, in unstable and overrated. If you bootstrap your data and try the RFE-CV, you’re likely to find little consistency in which features are selected. $\endgroup$
    – Dave
    Commented Mar 2, 2023 at 16:04
  • $\begingroup$ Thank you @Dave. Given the small sample size (n=160) and high-class imbalance (80:20), should I use repeated or nested cross-validation instead of a held-out test set (20% of data)? $\endgroup$
    – Dushi Fdz
    Commented Mar 2, 2023 at 16:32
  • $\begingroup$ That could be a reasonable next question to post. It warrants a full answer instead of just a comment. (Note, however, that such an answer is likely to mention the instability of feature selection.) $\endgroup$
    – Dave
    Commented Mar 2, 2023 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.