1
$\begingroup$

My data has 5 binary dependent variables, 9 categorical independent variables, and 3 continuous independent variables, with a sample size of 1232. The 5 dependent variables are just different ways of measuring enrollment persistence, specifically Next Subsession Persistence, Next Term Persistence, Next Year Persistence, Next Year Same Subsession Persistence, and Anytime Persistence. Two of the continuous independent variables may be correlated, since they are logically related - Incoming GPA and First Term GPA. Incoming GPA is high school GPA for the 100 first-time freshmen entering the program but transfer GPA for the 1132 transfer students entering the program.

I checked correlations of the continuous independent variables in SPSS and found that the Incoming GPA and First Term GPA correlation is only 0.115 with significance < 0.001, which I suspect is due to the large n or N, but it makes sense that they should be correlated.

Here's the question. Could multicollinearity be messing up the Logistic regression analysis here? Can I overcome it?

It's just that I'm finding that Logistic regression done separately for each of the 5 different independent variables is finding 2 of the categorical independent variables, which are Change of Major and Fulltime Enrollment, highly significant but in differing ways for the different types of persistence. Sometimes Change of Major is highly significant, sometimes Fulltime Enrollment is highly significant, sometimes both are. Also, neither Incoming GPA nor First Term GPA is significant in any of the models but I'm concerned about what may be a high correlation between them that I can't identify. I'm thinking that they must be correlated but their correlation is low and its high significance might be due to the high n.

Could multicollinearity be messing things up here? After reading several things off and on this site, I'm considering running a linear regression in SPSS to get multicollinearity VIFs even though what I'm going to ultimately perform will be logistic regression models. If I do that to reduce the number of independent variables in the final logistic regression analysis, I'm thinking it might be easier to interpret and the results might be more consistent. I can't find literature saying this is ever done but it seems like a good idea to me.

$\endgroup$
5
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/631126/… and my answer there. Basically multicollinearity plays out in the same way in logistic regression as in linear regression, so you might run a linear one to look at VIFs out of curiosity, but not for finally using it. $\endgroup$ Nov 15, 2023 at 10:33
  • 3
    $\begingroup$ Significance of correlation between "independent" variables is irrelevant as they are not assumed in any way to be independent. Quite a bit of correlation can actually be tolerated. $\endgroup$ Nov 15, 2023 at 10:44
  • 1
    $\begingroup$ Based on your description, my guess would be that multicollinearity is not affecting your results. If you run 1) a model with just incoming GPA and 2) a model with just 1st year GPA as predictor, how strongly each of them predicts your outcome in these reduced models? $\endgroup$
    – Sointu
    Nov 15, 2023 at 12:31
  • 1
    $\begingroup$ What is your research question? The interpretation of each coefficient depends on the other variables included in the model, so it isn't meaningful to compare the significance of a predictor in two models with different predictors. Any results you see could be due to confounding, conditioning on a mediator, or conditioning on a collider (yes, even if you are not doing a causal analysis). If your goal is prediction, why do you care about inference? If your goal is inference, what quantity (defined without respect to a model) are you doing inference on, and why? $\endgroup$
    – Noah
    Nov 15, 2023 at 17:39
  • $\begingroup$ The question is this: controlling for the other independent variables such as Incoming GPA, are Late Registration for classes and Change of Major associated with each type of enrollment persistence and if so, in which direction - increasing persistence or decreasing persistence. $\endgroup$ Nov 16, 2023 at 12:50

1 Answer 1

2
$\begingroup$

Multicollinearity might mess things up and might not. It really depends on what you're trying to do and what do you consider as risks...

If you want to use Logistic Regression for prediction (classification): Multicollinearity won't distort the model's output. You'll have the same numbers with features' removal or without it. One thing that's many times overlooked, is that multicollinearity might lead to numerical errors, as there's no limit to how big your coefficients may be. If one of your coefficients is at 1,000,000 the tiniest gap could distort your model's performance.

If you're using Logistic Regression as a descriptive or inference model: Multicollinearity is a serious issue. Your coefficients would probably not make any sense.

Overall, many scholars advice to remove features having perfect multicollinearity, as it may only harm your model. Reduction of features with a high VIF is one common way, there are plenty others.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer. I looked at the VIFs with linear regression using SPSS, and none of the VIFs was more than 3. Then I ran Chi-squared tests for associations between all the independent variables and each dependent variable, which is each type of persistence,. Then I removed the independent variables from each logistic regression model when the Chi-squared test didn't show association with the dependent variable in that model. $\endgroup$ Nov 16, 2023 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.