6
$\begingroup$

I was trying to determine the biasing constant in ridge regression when I came across a phenomenon that seems quite puzzling, to me at least. I let the GCV criterion choose a constant for me and then I got the Variance Inflation Factors of the new model by computing

$$ \left( \mathbf{R_{XX}} +c\mathbf{I} \right)^{-1} \mathbf{R_{XX}} \left( \mathbf{R_{XX}} +c\mathbf{I} \right)^{-1} $$

and extracting the diagonal elements of this matrix. What I found puzzling was the fact that these VIFs were very close to zero. It seems to me that that would require negative $R^2$s, no? I know that this can happen occasionally, for example in Regression Through the Origin, but I cannot quite justify it in this context.

I am wondering then, what does a VIF close to zero mean? Then, would my choice of this constant be acceptable or should I look for another solution that keeps the VIFs close to 1, as they ought to be in the absence of multicollinearity?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I have a similar question, only I'm doing plain old OLS and seeing some VIFs that are equaling exactly 0. Have you found any additional info on this elsewhere? $\endgroup$
    – sparc_spread
    Nov 28, 2015 at 20:36
  • 3
    $\begingroup$ @sparc_spread Someone told me a while back that you have to select the tuning so that the VIFs are close to 1, otherwise this will result in deflation. I tried that and it worked quite well. $\endgroup$
    – JohnK
    Nov 28, 2015 at 20:51
  • 1
    $\begingroup$ @sparc_spread How is it possible for OLS VIFs to equal zero? They are bounded below by 1. I suspect a misspecificaiton somewhere. $\endgroup$
    – JohnK
    Nov 28, 2015 at 21:28
  • 2
    $\begingroup$ You are absolutely right - I realized that they were for an indicator that was equaling 1 for all observations. So it was a misspecification for sure - the variable didn't belong in the model. $\endgroup$
    – sparc_spread
    Nov 28, 2015 at 21:29
  • 1
    $\begingroup$ If you consider the uncorrelated case $R_{XX}=I $, then the diagonal elements are $\frac{1}{(1+c)^2} $. This case will always have VIF small. $\endgroup$
    – probabilityislogic
    Apr 23, 2018 at 23:47

3 Answers 3

Reset to default
2
$\begingroup$

I would like to suggest that you calculate the diagonal elements of matrix directly.

It is assumed that the design matrix is centered and scaled.

We can adopt the eigen value decomposition $R_{XX}=X'X=T\Lambda T'$.

$\begin{align} (R_{XX}+cI)^{-1}R_{XX}(R_{XX}+cI)^{-1}&=(R_{XX}+cI)^{-1}(R_{XX}+cI)(R_{XX}+cI)^{-1}-c(R_{XX}+cI)^{-1}(R_{XX}+cI)^{-1}\\ &=(R_{XX}+cI)^{-1}-c(R_{XX}+cI)^{-1}(R_{XX}+cI)^{-1} \\ &=(T\Lambda T'+cTT')^{-1}-c(T\Lambda T'+cTT')^{-1}(T\Lambda T'+cTT')^{-1}\\ &=T\left( (\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1} \right)T' \end{align}$

The matrix $ (\Lambda+cI)^{-1}$ is a diagonal matrix that its $i$th element is $\frac{1}{\lambda_i+c}$.

So the matrix $(\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1}$ is also a diagonal matrix and its ith element is $\frac{\lambda_i}{(\lambda_i+c)^2}$.

In OLS, it is known that vif values are the diagonal elements of the matrix $T\Lambda^{-1}T'$. Comparing this $\Lambda^{-1}$ matrix with the corresponding of ridge$(\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1}$, every diagonal elements of the ridge case are deflated by the factor $\frac{\lambda_i^2}{(\lambda_i+c)^2}$.

I guess now we can conclude the bigger the ridge constant, we would get the more deflated VIFs.

I am not a native English speaker. Please don't mind my awkward expressions and it would be nice of you if you correct my grammar errors. Thank you.

Improve this answer
$\endgroup$
1
  • $\begingroup$ And I guess the form $\frac{1}{1-R_i^2}$ is not the case. $\endgroup$
    – KDG
    Sep 4, 2018 at 1:15
1
$\begingroup$

I was wrestling with this issue myself, and then discovered this article by Garcia et al., where they show how traditional definition of the VIF does in fact lead to values less than unity in the case of ridge regression. They subsequently propose an alternate definition, involving the ridge parameter $k$, which leads to VIFs bounded which are bounded from below by one. In the case of a two estimators with correlation $\rho$, if the number of observations is $n$, the expression for the VIF is $$ VIF(k, n)=\frac{((n + 2)(1 + k) - k)^2}{(n + 2)^2((1 + k)^2 - \rho^2) - 2(n + 2)k(1 + k-\rho)}\ . $$

Improve this answer
$\endgroup$
0
$\begingroup$

Recall that you are minimizing $$ \|Ax -y\|^2 + c\|x-0\|^2 $$ So as $c$ increases $x\to 0$ i.e. the bias is such that your estimator approches $0$ as $c$ increases. So when your VIFs are approaching one then stop increasing $c$, that is probably your optimal $c$.

Reference: p. 434 in Applied Statistical Models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.