The Variance Inflation Factor (VIF) is defined to be:

$$VIF_p = \frac{1}{1-R_p^2}$$

where $R_p^2$ is the $R^2$ calculated when $X_p$ is the dependent variable, and all the other variables are independent.

Similarly, in Factor Analysis, the Squared Multiple Correlation (SMC) is the $R^2$ of each variable regressed on all the other variables.

In both cases there seems to be a trick where instead of calculating the actual regressions, one can use the inverse of the correlation matrix. The VIFs are the diagonal of that inverted matrix, and the SMCs are $1-1/diag$.

How can one show this?

  • $\begingroup$ Not sure what your question really asks, but maybe en.wikipedia.org/wiki/Variance_inflation_factor#Definition helps? $\endgroup$
    – Zhanxiong
    Commented Apr 29, 2023 at 18:00
  • $\begingroup$ @Zhanxiong nope. Doesn't help at all. I'm asking about the trick of calculating VIF from the correlation matrix, which doesn't appear at all at the wikipedia page. $\endgroup$ Commented Apr 30, 2023 at 21:17
  • $\begingroup$ Check, e.g., github.com/statsmodels/statsmodels/issues/… $\endgroup$ Commented Apr 30, 2023 at 21:19
  • $\begingroup$ If by correlation matrix, you meant the matrix $X^TX$, then that wikipedia page tells you $VIF_j = (n - 1)\operatorname{var}(X_j)[(X^TX)^{-1}]_{jj}$. That is, a multiple of the $j$-th diagonal entry of the inverse matrix of $X^TX$. $\endgroup$
    – Zhanxiong
    Commented Apr 30, 2023 at 21:23
  • 1
    $\begingroup$ $X^TX$ is not a correlation matrix, unless you center and scale the data. I checked now and it does seem that the VIF/$R^2$ stays the same, meaning that the RSS/TSS is the same for both normalized and unnormalized data... So, the only missing part for me is what happens to the RSS when using normalized data vs. regular data. It seems to absorb the TSS into it. But I would like to see this algebraically, preferably with matrix notation. $\endgroup$ Commented May 1, 2023 at 11:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.