2
$\begingroup$

I'm having trouble deciding what to do when two important variables (important to my research questions) are seemingly too related, but both need to be included in a full model (this is a longitudinal study where we repeatedly sample around the same time 2x each year, i.e. every season). I also have a related and broader conceptual question on the difference between prediction and multicollinearity (linear modeling and VIF). If my goal was to address potential multicollinearity, and/or covariate redundancy, in a full multiple linear regression model, and I do this by excluding variables that are too similar, how/why is running a simple linear model of two "trouble" covariates different from running VIF on the full model (why is prediction giving me a different answer than my check multicollinearity)? The two ideas seem similar: "how is one variable related to another?" (I'm trying to resolve my obvious ignorance of the difference between these two concepts).

If these were too continuous variables, I would simply check their correlation/collinearity and pick one, then be done, but one being categorical has thrown me down this rabbit hole. As an example, I have created a categorical variable "Season" and a continuous variable "water temperature" (variables which are both important to my research questions and representing the sample design, but are probably too closely related) to represent my actual data. There is a definite seasonal component to water temp., and if I model one with the other, that model tells me Season is a statistically significant variable for predicting water temperature. However, why wouldn't VIF tell me season and water temperature are too highly correlated in the same model together? My gut screams these are completely different things, but I don't quite understand why both wouldn't give me the same answer, at least in different ways. Shouldn't both tell me one is a "good" predictor of the other (VIF should also be high)? Both are significant in this hypothetical full model, and taking one away makes the remainder insignificant (a multicollinearity issue, no? So, why didn't VIF pick that up?). Thank you in advance and my apologies if this sounds like nonsense.

Thinking on it further lead me to posit that "high" VIF values are slightly subjective and that my results are indeed saying the two variables are related, but not so badly that it's a problem (?)

> # Water temperature (actual means and SD from my data):

> df <- as.data.frame(rnorm(50, mean = 23.8, sd = 2.60)) 
> df2 <- as.data.frame(rnorm(50, mean = 30.2, sd = 1.62))

> names(df)[1] <- "water_temp"
> names(df2)[1] <- "water_temp"

> df3 <- rbind(df, df2)

> df3$season <- rep(c("DRY", "WET"), each=50)

> # Other random variables:
> df3$v1 <- sample(0:3000, 100, replace = TRUE)

> df3$v2 <- rnorm(100, mean = 25, sd = 5)

> # Dependent variable:
> x <- rnorm(100, mean=0, sd=1)

> df3$x <- x

> mod <- lm(x ~ water_temp + season + v1 + v2, data=df3)

> vif.cat.data <- check_collinearity(mod) 
> vif.cat.data
# Check for Multicollinearity

Low Correlation

       Term  VIF       VIF 95% CI Increased SE Tolerance Tolerance 95% CI
 water_temp 3.55 [2.65,     4.93]         1.88      0.28     [0.20, 0.38]
     season 3.54 [2.65,     4.92]         1.88      0.28     [0.20, 0.38]
         v1 1.02 [1.00,    61.86]         1.01      0.98     [0.02, 1.00]
         v2 1.01 [1.00, 1.62e+05]         1.01      0.99     [0.00, 1.00]



> mod2 <- lm(water_temp ~ season, data = df3)
> anova(mod2)
Analysis of Variance Table

Response: water_temp
          Df Sum Sq Mean Sq F value    Pr(>F)    
season     1 998.73  998.73  246.48 < 2.2e-16 ***
Residuals 98 397.09    4.05                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Evidence of multicollinearity?

> summary(mod)

Call:
lm(formula = x ~ water_temp + season + v1 + v2, data = df3)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.4745 -0.6249  0.2011  0.6579  2.0758 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)   
(Intercept) -4.101e+00  1.251e+00  -3.277  0.00147 **
water_temp   1.387e-01  4.973e-02   2.790  0.00638 **
seasonWET   -1.044e+00  3.713e-01  -2.811  0.00600 **
v1          -5.196e-05  1.122e-04  -0.463  0.64428   
v2           4.000e-02  1.836e-02   2.178  0.03187 * 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9866 on 95 degrees of freedom
Multiple R-squared:  0.1331,    Adjusted R-squared:  0.0966 
F-statistic: 3.647 on 4 and 95 DF,  p-value: 0.008281


# Remove "season"
> mod <- lm(x ~ water_temp + v1 + v2, data=df3)
> summary(mod)

Call:
lm(formula = x ~ water_temp + v1 + v2, data = df3)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.4078 -0.7440  0.1653  0.6890  2.2550 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)  
(Intercept) -1.553e+00  8.934e-01  -1.738   0.0854 .
water_temp   2.078e-02  2.763e-02   0.752   0.4540  
v1          -5.288e-05  1.161e-04  -0.455   0.6499  
v2           4.443e-02  1.894e-02   2.346   0.0211 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.021 on 96 degrees of freedom
Multiple R-squared:  0.061, Adjusted R-squared:  0.03165 
F-statistic: 2.079 on 3 and 96 DF,  p-value: 0.1081


# Remove water_temp

> mod <- lm(x ~ season + v1 + v2, data=df3)
> mod2 <- lm(water_temp ~ season, data = df3)
> summary(mod)

Call:
lm(formula = x ~ season + v1 + v2, data = df3)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.53825 -0.71103  0.07287  0.74687  2.18310 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)  
(Intercept) -9.134e-01  5.284e-01  -1.729   0.0871 .
seasonWET   -1.698e-01  2.062e-01  -0.823   0.4123  
v1          -2.873e-05  1.158e-04  -0.248   0.8045  
v2           4.315e-02  1.896e-02   2.276   0.0251 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.021 on 96 degrees of freedom
Multiple R-squared:  0.06209,   Adjusted R-squared:  0.03278 
F-statistic: 2.119 on 3 and 96 DF,  p-value: 0.1029
$\endgroup$

2 Answers 2

8
$\begingroup$

For one thing, regression coefficients are about the relationship between independent variables and the dependent variable. VIF is only about the independent variable.

Your section starting with "Evidence of multicollnearity?" has no evidence for or against that I could see.

My dissertation (way back in 1999!) showed that condition indexes are a better way to diagnose collinearity than VIF. (If the link goes bad, just Google my name and "Monte Carlo").

For your specific problem, season and temperature are not both needed, unless season adds something that temperature does not. But they are likely to be colinear.

Finally, you imply that correlation is a measure of collinearity. While there is a relationship, it is possible to have high collinearity with low correlations. Suppose that there are 10 uncorrelated variables and one that is the sum of those 10. None of the correlations will be very high but, if the summation has no error, colinearity will be perfect.

$\endgroup$
8
  • 1
    $\begingroup$ Thanks for the authoritative answer on indices for detecting collinearity. I have never seen codition indices being recommended anywhere. Is there some (lightweight) R package that implements them? $\endgroup$
    – cdalitz
    Oct 16, 2023 at 19:31
  • $\begingroup$ The perturb package has them. So does the olsrr package. $\endgroup$
    – Peter Flom
    Oct 16, 2023 at 22:52
  • 1
    $\begingroup$ Suspect last 3 words are a typo for "collinearity will be perfect"! $\endgroup$
    – Silverfish
    Oct 17, 2023 at 5:17
  • 1
    $\begingroup$ I am also a big fan of the approach proposed by Belsley, Kuh & Welsch. Belsley (1991) is an accessible treatment for the practitioner who would like to understand how this approach can actually be used, rather than the statistician who needs to understand why it works. It is unfortunately a bit more complex than VIFs. $\endgroup$ Oct 17, 2023 at 7:32
  • $\begingroup$ Just tried out condition indices on the dataset nhospital from the R package GLMsData, which has highly correlated predictors. The highest condition index for the model MainHours ~ Cases + Eligible + OpRooms is 16. According to Beasley (1991), this means that there is no collinearity to worry about. OTOH, VIFs of 16 and 13 are reported, which indicate strong multicollinearity. In this case, VIFs seem to be better indicators. What is going wrong with condition indices in this very simple example with obvious collinearities? $\endgroup$
    – cdalitz
    Nov 21, 2023 at 12:55
3
$\begingroup$

It seems you are confused both about beta and VIF:

  • Beta gives some strength in the relationship, indicating how much the target move with the feature, but it is not really related to the quality of the predictor / model. The quality of a predictor / model would be evaluated by looking at how much of the variance is explained by the predictor instead of noise, that is using R2. (Another way to see this is considering rescaling features: rescaling change beta but should not generally change the quality of the model, hence beta is not really linked to quality of the model).

  • VIF relates to the correlation of one variables to the others. That is how the remaining features could explain one given feature. It generally gives an idea on how you can remove that feature (because it can be explained by the others), but not the link between that feature and the target. It is a bit dangerous as the target may lie in the difference of features. The edge exemple is trying to predict y from X and X+y. A model would give: y = X + y - X. But VIF could remove X + y before modelling.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.