3
$\begingroup$

For example, if we have the linear regression model:

$$E(y) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 $$

where $ x_1 =\begin{cases} 1 & \mbox{if level 2} \\ 0 & \mbox{otherwise} \end{cases}$ and $x_2, x_3 $ are quantitative.

When checking for multicollinearity, we typically compute the linear regression models for each independent variable as a function of the remaining independent variables:

\begin{align} E[x_1] &= \alpha_0 + \alpha_2 x_2 + \alpha_3 x_3 \\ E[x_2] &= \alpha_0 + \alpha_1 x_1 + \alpha_3 x_3 \\ E[x_3] &= \alpha_0 + \alpha_2 x_2 + \alpha_2 x_2 \end{align}

And from there, we derive the VIF for each term from $R^{2}_{i}\, \forall i \in \{1, 2, 3\} $ (just the $R^2$ for the models above).

My problem is with $ E[x_1]$ which is possibly non-sensical, or at least counter-intuitive, as the possible values of $ x_1 $ are only $0$ or $1$. Of course, computing $ E[x_1]$ does make sense mathematically, but is this exactly what we need? Please explain.

Is there a way to calculate the $VIF$ for a categorical variable to check for how it is affected by multicollinearity?

(And an extra thank you if you know of an library in R which calculates the VIF for a linear regression model. Maybe the vif function?)

$\endgroup$
  • 1
    $\begingroup$ The regression of $x_1$ against $x_2$ and $x_3$ might not make any sense, but it's perfectly well defined mathematically: and its $R^2$ is precisely what the VIF needs you to compute. See car:::vif.default for working code. $\endgroup$ – whuber Dec 9 '15 at 20:54
  • $\begingroup$ Is there any restrictions to the interpretation of the $R^2$ and $VIF$ for the model having $x_1$ against $x_2$ and $x_3$? What I am trying to get at is why, if we normally would not fit a linear regression model to a categorical response, is it still valid to calculate the $VIF$ this way? $\endgroup$ – gbrlrz017 Dec 10 '15 at 0:57
4
$\begingroup$

The function you requested comes in the package {car} in R.

I tried to figure it out running some regression models using the mtcars package in R.

Evidently, I can get the VIF both using the function and manually, when the regressor is a continuous variable:

require(car)
attach(mtcars)

fit1 <- lm(mpg ~ wt + hp + disp)     # The model we want.
fit_wt <- lm(wt ~ hp + disp)         # Regressing wt against other regressors.
rsq_wt <- summary(fit_wt)$r.square   # Detecting the R square of the model
(v_wt <- 1/(1 - (rsq_wt)))           # Actual formula for VIF
vif(fit1)                            # R built-in function

Now for the real question, here is what I find. Let's say that your regressor is am, which corresponds to the categorical variable for the type of transmission of the car (automatic versus manual).

Ordinarily, you would fit a model such as:

fit2 <- lm(mpg ~ wt + disp + as.factor(am))

The problem is that if you try now to get the VIF for am by just reshuffling the regressors you get an error message:

fit_am <- lm(as.factor(am) ~ wt + disp)
Warning messages:
1: In model.response(mf, "numeric") :
  using type = "numeric" with a factor response will be ignored
2: In Ops.factor(y, z$residuals) : - not meaningful for factors

Game over? Not quite... Look what happens if I treat am as continuous:

> fit2 <- lm(mpg ~ wt + disp + as.factor(am))
> fit_am <- lm(am ~ wt + disp)
> rsq_am <- summary(fit_am)$r.square
> (v_am <- 1/(1 - (rsq_am)))
[1] 1.931264
> vif(fit2)
           wt          disp as.factor(am) 
     5.939675      4.752561      1.931264 

We get the same value manually as with the R built-in function vif.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ah! It is good to see how the VIF is calculated in R. I am still a bit confused as to if we should should interpret VIF as it is calculated in R (with assumption that the categorical variable is continuous). In practice, it seems like we wouldn't be able to get anything useful out of such a model--at least not for prediction. I'm wondering if the $R^2$ and VIF for the categorical variable still retain their meaning of measuring collinearity despite this. $\endgroup$ – gbrlrz017 Dec 10 '15 at 1:34
  • $\begingroup$ I really don't have answers for your questions. I was happy to hack away the workings of the R function because your question left me intrigued. $\endgroup$ – Antoni Parellada Dec 10 '15 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.