I have learnt that using vif() method of car package, we can compute the degree of multicollinearity of inputs in a model. From wikipedia, if the vif value is greater than 5 then we can consider that the input is suffering from multicollinearity problem. For example, I have developed a linear regression model using lm() method and vif() gives as follows. As we can see, the inputs ub, lb, and tb are suffering from multicollinearity.

     tb        ub        lb          ma     ua        mb         sa     sb 
 7.929757 50.406318 30.826721  1.178124  1.891218  1.364020  2.113797  2.357946

In order to avoid the multicollinearity problem and thus to make my model more robust, I have taken interaction between ub and lb, and now vif table of new model is as follows:

   tb     ub:lb      ma       mb      sa        sb     ua
1.763331 1.407963 1.178124 1.327287 2.113797 1.860894 1.891218

There is no much difference in R^2 values and as well as there is no much difference in the errors from one-leave-out CV tests in both the above two cases.

My questions are:

  1. Is it fine to avoid the multicollinearity problem by taking the interaction as shown above?

  2. Is there any nicer way to present multicollinearity problem compared with the above vif method results.

Please provide me your suggestions.


  • $\begingroup$ Is your model log-log regression (interaction as the restriction as ok with them)? In wiki there is a more complete article regarding multicollinearity issue en.wikipedia.org/wiki/Multicollinearity Also I suggest to explore the multicollinearity tag on this site, it seems to be a frequently asked question. $\endgroup$ Commented Jul 27, 2011 at 9:41
  • $\begingroup$ @Dmitrij Celov: Thank you. No, my model is simple linear regression without logs... $\endgroup$
    – samarasa
    Commented Jul 27, 2011 at 15:38
  • $\begingroup$ I think (statisticalaid.com/…) this can help you. $\endgroup$ Commented Oct 11, 2021 at 16:39

2 Answers 2


You seem to include the interaction term ub:lb, but not ub and lb themselves as separate predictors. This would violate the so-called "principle of marginality" which states that higher-order terms should only include variables present in lower-order terms (Wikipedia for a start). Effectively, you are now including a predictor that is just the element-wise product of ub and lb.

$VIF_{j}$ is just $\frac{1}{1-R_{j}^{2}}$ where $R_{j}^{2}$ is the $R^{2}$ value when you run a regression with your original predictor variable $j$ as criterion predicted by all remaining predictors (it is also the $j$-th diagonal element of $R_{x}^{-1}$, the inverse of the correlation matrix of the predictors). A VIF-value of 50 thus indicates that you get an $R^{2}$ of .98 when predicting ub with the other predictors, indicating that ub is almost completely redundant (same for lb, $R^{2}$ of .97).

I would start doing all pairwise correlations between predictors, and run the aforementioned regressions to see which variables predict ub and lb to see if the redundancy is easily explained. If so, you can remove the redundant predictors. You can also look into ridge regression (lm.ridge() from package MASSin R).

More advanced multicollinearity diagnostics use the eigenvalue-structure of $X^{t}X$ where $X$ is the design matrix of the regression (i.e., all predictors as column-vectors). The condition $\kappa$ is $\frac{\sqrt{\lambda_{max}}}{ \sqrt{ \lambda_{min}}}$ where $\lambda_{max}$ and $\lambda_{min}$ are the largest and smallest ($\neq 0$) eigenvalues of $X^{t}X$. In R, you can use kappa(lm(<formula>)), where the lm() model typically uses the standardized variables.

Geometrically, $\kappa$ gives you an idea about the shape of the data cloud formed by the predictors. With 2 predictors, the scatterplot might look like an ellipse with 2 main axes. $\kappa$ then tells you how "flat" that ellipse is, i.e., is a measure for the ratio of the length of largest axis to the length of the smallest main axis. With 3 predictors, you might have a cigar-shape, and 3 main axes. The "flatter" your data cloud is in some direction, the more redundant the variables are when taken together.

There are some rules of thumb for uncritical values of $\kappa$ (I heard less than 20). But be advised that $\kappa$ is not invariant under data transformations that just change the unit of the variables - like standardizing. This is unlike VIF: vif(lm(y ~ x1 + x2)) will give you the same result as vif(lm(scale(y) ~ scale(x1) + scale(x2))) (as long as there are not multiplicative terms in the model), but kappa(lm(y ~ x1 + x2)) and kappa(lm(scale(y) ~ scale(x1) + scale(x2))) will almost surely differ.

  • $\begingroup$ Regarding the first note: for multivariate model it seems ok to leave only interaction term, since the latter just means you restrict the coefs for ub lb ($a\cdot \log ub + a \cdot \log lb = a \log ub\cdot lb$), but for a general linear regression model it is as you pointed out (+1)^_^ By the way lb - lower bound, ub - upper bound?! if so make the difference of two $ub - lb$, it is probably the better transformation. $\endgroup$ Commented Jul 27, 2011 at 10:52
  • $\begingroup$ @Dmitrij: No, ub + lb + sb = 100. $\endgroup$
    – samarasa
    Commented Jul 27, 2011 at 16:59
  • $\begingroup$ Is there any nice graphical representation for multicollinearity? PLease suggest me if there is any method in R for thi. $\endgroup$
    – samarasa
    Commented Jul 27, 2011 at 17:00
  • 2
    $\begingroup$ @kkp The only ways I can think of right now are: pairs() for scatterplot matrices showing all pairwise distribution of the predictors. Package scatterplot3d can be used to visualize the scatterplot of 3 variables. See this answer for drawing the ellipse whose axis-ratio equals $\kappa^{2}$ in the bivariate case. You might also check package perturb which offers additional multicollinearity diagnostics, maybe graphics. $\endgroup$
    – caracal
    Commented Jul 27, 2011 at 18:41

You should also consider P-value during variable consideration.

  1. If P-Value is very low (p<0.05) then and VIF is high (>5) then may be you should consider other insignificant variables. And rebuild your model.
  2. If you have P-value and VIF both high then this variable will be insignificant.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.