5
$\begingroup$

I was trying to determine the biasing constant in ridge regression when I came across a phenomenon that seems quite puzzling, to me at least. I let the GCV criterion choose a constant for me and then I got the Variance Inflation Factors of the new model by computing

$$ \left( \mathbf{R_{XX}} +c\mathbf{I} \right)^{-1} \mathbf{R_{XX}} \left( \mathbf{R_{XX}} +c\mathbf{I} \right)^{-1} $$

and extracting the diagonal elements of this matrix. What I found puzzling was the fact that these VIFs were very close to zero. It seems to me that that would require negative $R^2$s, no? I know that this can happen occasionally, for example in Regression Through the Origin, but I cannot quite justify it in this context.

I am wondering then, what does a VIF close to zero mean? Then, would my choice of this constant be acceptable or should I look for another solution that keeps the VIFs close to 1, as they ought to be in the absence of multicollinearity?

$\endgroup$
  • 1
    $\begingroup$ I have a similar question, only I'm doing plain old OLS and seeing some VIFs that are equaling exactly 0. Have you found any additional info on this elsewhere? $\endgroup$ – sparc_spread Nov 28 '15 at 20:36
  • 3
    $\begingroup$ @sparc_spread Someone told me a while back that you have to select the tuning so that the VIFs are close to 1, otherwise this will result in deflation. I tried that and it worked quite well. $\endgroup$ – JohnK Nov 28 '15 at 20:51
  • 1
    $\begingroup$ @sparc_spread How is it possible for OLS VIFs to equal zero? They are bounded below by 1. I suspect a misspecificaiton somewhere. $\endgroup$ – JohnK Nov 28 '15 at 21:28
  • 2
    $\begingroup$ You are absolutely right - I realized that they were for an indicator that was equaling 1 for all observations. So it was a misspecification for sure - the variable didn't belong in the model. $\endgroup$ – sparc_spread Nov 28 '15 at 21:29
  • 1
    $\begingroup$ If you consider the uncorrelated case $R_{XX}=I $, then the diagonal elements are $\frac{1}{(1+c)^2} $. This case will always have VIF small. $\endgroup$ – probabilityislogic Apr 23 '18 at 23:47
2
$\begingroup$

I would like to suggest that you calculate the diagonal elements of matrix directly.

It is assumed that the design matrix is centered and scaled.

We can adopt the eigen value decomposition $R_{XX}=X'X=T\Lambda T'$.

$\begin{align} (R_{XX}+cI)^{-1}R_{XX}(R_{XX}+cI)^{-1}&=(R_{XX}+cI)^{-1}(R_{XX}+cI)(R_{XX}+cI)^{-1}-c(R_{XX}+cI)^{-1}(R_{XX}+cI)^{-1}\\ &=(R_{XX}+cI)^{-1}-c(R_{XX}+cI)^{-1}(R_{XX}+cI)^{-1} \\ &=(T\Lambda T'+cTT')^{-1}-c(T\Lambda T'+cTT')^{-1}(T\Lambda T'+cTT')^{-1}\\ &=T\left( (\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1} \right)T' \end{align}$

The matrix $ (\Lambda+cI)^{-1}$ is a diagonal matrix that its $i$th element is $\frac{1}{\lambda_i+c}$.

So the matrix $(\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1}$ is also a diagonal matrix and its ith element is $\frac{\lambda_i}{(\lambda_i+c)^2}$.

In OLS, it is known that vif values are the diagonal elements of the matrix $T\Lambda^{-1}T'$. Comparing this $\Lambda^{-1}$ matrix with the corresponding of ridge$(\Lambda+cI)^{-1}-c (\Lambda+cI)^{-1} (\Lambda+cI)^{-1}$, every diagonal elements of the ridge case are deflated by the factor $\frac{\lambda_i^2}{(\lambda_i+c)^2}$.

I guess now we can conclude the bigger the ridge constant, we would get the more deflated VIFs.

I am not a native English speaker. Please don't mind my awkward expressions and it would be nice of you if you correct my grammar errors. Thank you.

$\endgroup$
  • $\begingroup$ And I guess the form $\frac{1}{1-R_i^2}$ is not the case. $\endgroup$ – KDG Sep 4 '18 at 1:15
0
$\begingroup$

Recall that you are minimizing $$ \|Ax -y\|^2 + c\|x-0\|^2 $$ So as $c$ increases $x\to 0$ i.e. the bias is such that your estimator approches $0$ as $c$ increases. So when your VIFs are approaching one then stop increasing $c$, that is probably your optimal $c$.

Reference: p. 434 in Applied Statistical Models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.