3
$\begingroup$

In this question, the OP runs a “chunk test” and has a linear relationship between a variable in the restricted model and the variables in the chunk.

If this were a chunk of just one variable, that linear relationship would manifest through the variance-inflation factor and inflate the p-value of the t-test of that one variable. When we test a chunk of multiple variables, something similar should happen, but what is the math of how the variance-inflation factor appears in the F-test?

For one example of how this could happen, we could run an ANCOVA but have the factor variable being tested be somewhat predictive of our covariate. However, I am interested in more generality. The restricted model can have multiple variables, and the chunk could be any chunk of variables (not just a categorical variable like the ANCOVA example).

EDIT

EdM gave a nice answer to what I posted, but to clarify, my interest is more about relationships between in-chunk and out-of-chunk variables than it is about relationships within the chunk being tested.

$\endgroup$
2
  • 3
    $\begingroup$ A "chunk" can be highly significant while variables within the chunk are insignificant, due to multicollinearity of variables within the chunk. So variance inflation does not seem to be an issue for the chunk test, despite collinearity within the chunk. As an example, there is perfect multicollinearity between variables in the full dummy representation of a factor, but the partial F test is not affected. On the other hand, if there is multicollinearity between chunks, there will definitely be a variance inflation issue. Seems like interesting math involving a cross- correlation matrix. $\endgroup$ Commented Dec 2, 2022 at 12:55
  • 1
    $\begingroup$ @BigBendRegion Great comments. Much of the interesting math to which you refer was done in 1980 by Belsley, Kuh, & Welsch, Regression diagnostics. They focus on finding groups of variables that might contribute individually to multicollinearity, yet be relatively uncorrelated between groups. $\endgroup$
    – whuber
    Commented Jan 18, 2023 at 20:17

1 Answer 1

2
+50
$\begingroup$

The variance-inflation factor for an individual predictor in an ordinary least-squares model is based on the multiple $R^2$ for its linear regression against all the other predictors.* Any correlation with the other predictors inflates the variance of its coefficient estimate beyond an uncorrelated situation.

The quadratic form involved in a simultaneous "chunk" test on multiple coefficients, in contrast, means that correlations between predictors can either increase or decrease the value of the test statistic depending on the signs of correlations and of coefficient estimates.

A simple example with two predictors illustrates the principle, with the additional simplification that both predictors have the same variance inflation factor. Start with the usual ordinary least squares setup, with continuous outcome $y$ and normally distributed error $\epsilon$:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon.$$

Center and scale each of $x_1$ and $x_2$ to zero mean and unit standard deviation. That simplifies the design matrix $X$ so that:

$$X^TX=N \begin{pmatrix}1 & 0 &0\\0&1&r\\0& r& 1 \end{pmatrix}$$

where $N$ is the number of observations and $r$ is the sample correlation coefficient between $x_1$ and $x_2$. The covariance matrix of coefficient estimates, with $\hat\sigma^2$ the estimate of error variance, is:

$$\hat\sigma^2 (X^TX)^{-1} = \frac{\hat\sigma^2}{N} \begin{pmatrix}1 & 0 &0\\0&\frac{1}{1-r^2}&\frac{-r}{1-r^2}\\0& \frac{-r}{1-r^2}& \frac{1}{1-r^2} \end{pmatrix} .$$

The diagonal elements are the variances of the individual coefficient estimates, with variance inflation $1/(1-r^2)$. The covariance between $\hat\beta_1$ and $\hat\beta_2$ is opposite in sign to $r$. The Wald statistic for the hypothesis $\beta_1=0$ is then:

$$\frac{N \left( 1-r^2 \right) \hat\beta_1^2}{\hat\sigma^2},$$

with a corresponding result for $\beta_2$. Higher values of $r^2$ lower the values of test statistics on the individual coefficients, regardless of the sign of $r$. With more predictors, $1/(1-r^2)$ would be replaced by variance inflation factors in single-coefficient tests.

A Wald "chunk" test on the joint hypothesis that $\beta_1=\beta_2=0$ is based on the quadratic form between the vector $\begin{pmatrix}\hat\beta_1 & \hat\beta_2\end{pmatrix}$ and the inverse of the corresponding submatrix of the coefficient covariance matrix (last two columns and rows, in this example). The inverse of that submatrix is simply related to the corresponding submatrix of $X^TX$, giving:

$$\begin{pmatrix} \hat\beta_1 & \hat\beta_2\end{pmatrix} \frac{N}{\hat\sigma^2} \begin{pmatrix} 1 & r \\r&1 \end{pmatrix} \begin{pmatrix}\hat\beta_1 \\\hat\beta_2 \end{pmatrix}= \frac{N}{\hat\sigma^2} \left( \hat\beta_1^2 + 2 r \hat\beta_1 \hat\beta_2 + \hat\beta_2^2\right).$$

A positive correlation $r$ between $x_1$ and $x_2$, other things being equal, will increase the value of the "chunk" test statistic over the uncorrelated situation if $\hat\beta_1$ and $\hat\beta_2$ have the same sign, as will a negative correlation if the coefficient estimates have opposite signs.


*There's a generalized variance-inflation factor for other forms of regression; see this page.

$\endgroup$
5
  • $\begingroup$ Does this imply that the correlation between the features in the chunk being tested plays a big role in the hypothesis test? $\endgroup$
    – Dave
    Commented Feb 14, 2023 at 13:40
  • $\begingroup$ @Dave yes. The correlations among features lead to off-diagonal entries of the coefficient covariance matrix, as shown in the third equation. Those off-diagonal entries help determine both the standard errors of model predictions, from the formula for the variance of a weighted sum of variables, and the Wald statistic evaluated in a "chunk" test, as in the last equation of the answer. Software typically reports only the individual coefficient standard errors, the square roots of the diagonal entries, unless you ask. $\endgroup$
    – EdM
    Commented Feb 14, 2023 at 15:36
  • $\begingroup$ Unbelievable...the VIF makes its way into the chunk test of both variables simultaneously, too. Your math makes sense, and a quick simulation gives a nice visual. I think I did not explain myself well in the original question, however. My interest is less in how multicollinearity within the chunk impacts the chunk test and more in how a relationship between the in-chunk and out-of-chunk variables affects the test. I have edited the OP to clarify. $\endgroup$
    – Dave
    Commented Feb 14, 2023 at 16:14
  • $\begingroup$ Unbelievable...the VIF makes its way into the chunk test of both variables simultaneously, too. Your math makes sense, and a quick simulation gives a nice visual. I think I did not explain myself well in the original question, however. My interest is less in how multicollinearity within the chunk impacts the chunk test and more in how a relationship between the in-chunk and out-of-chunk variables affects the test. I have edited the OP to clarify. // But what about Harrell’s comment in the link that a chunk test does not suffer from multicollinearity issues within the chunk? $\endgroup$
    – Dave
    Commented Feb 14, 2023 at 16:20
  • $\begingroup$ @Dave for a quick simple view, think of the $\beta_1=0$ hypothesis as a "chunk" test (with only 1 piece in the chunk). Then with $\beta_2$ outside that "chunk", the variance for the "chunk" test restricted to $\beta_1$ is inflated by the VIF. That should generalize to higher-dimensional "chunks." Harrell's comment is illustrated in the last equation for the usual multicollinearity case of a "chunk" test combining positively correlated predictors associated in the same direction with outcome: their correlation increases the Wald statistic for the "chunk" test. $\endgroup$
    – EdM
    Commented Feb 14, 2023 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.