# How to perform a joint significance test for multiple coefficients across several regression models?

I have 10 regression models, each regressing a different dependent variable $$y_i$$ (for $$i = 1, 2, \ldots, 10$$) on the same independent variable $$x$$. Each model produces a coefficient for $$x$$, and I am interested in testing the joint significance of all these 10 coefficients together.

Specifically, my models are:

$$y_1 = \beta_{1x} x + \epsilon_1$$

$$y_2 = \beta_{2x} x + \epsilon_2$$

$$\vdots$$

$$y_{10} = \beta_{10x} x + \epsilon_{10}$$

I would like to know the best approach to test if all $$\beta_{ix}$$ (for $$i = 1, 2, \ldots, 10$$) are jointly significantly different from zero.

The context is that I have a technology shock x that impacts different regions and varies over time, expected to affect industry y's employment in those regions. I have already established this link. However, to demonstrate that my shock is not correlated with other latent factors at the location level, I want to show that my shock cannot predict employment in other supposedly unrelated industries. That's why I have included these other industries as placebo industries ($$y_1$$ to $$y_{10}$$).

My outcome variables (different y's) represent employment in different industries across different regions, which have different magnitudes and thus likely different variances. I have region and time-fixed effects in my models, and the unit of observation is location-time, if that helps.

Is there a statistical method or a specific test that I can use for this purpose? I am familiar with F-test but my understanding is F-test is used when models are nested which doesn't seem to be the case for me.

Thank you!

• What do you mean by jointly significantly different from zero? For example, why not simply correct the $10$ corresponding $p$-values for multiple testing? Is it necessary that all are significantly non-zero? Commented Jul 1 at 17:12
• @FransRodenburg Maybe I'm using the wrong terminology, so a bit of context might be helpful. Each regression in my analysis includes a placebo outcome variable, meaning that ideally, I expect all coefficients to be zero. Each of these regressions provides me with a p-value to test whether each coefficient is statistically different from zero. However, I am looking to perform a joint test to determine whether all the coefficients are jointly significant or not. That is, I eventually want to have a p-value for the hypothesis that all coefficients are zero. Commented Jul 1 at 17:31
• @whuber I don't think I can make that assumption. My outcome variables (different y's) represent employment in different sectors across different regions, which have different magnitudes and thus likely different variances. Commented Jul 1 at 17:50
• Note that a significance test tests a single null hypothesis (although this can be formally pieced together out of several hypotheses), and the term "significant" refers to the outcome of the test rather than the underlying true parameters. This means that if you want to run a single test, it doesn't make sense to claim that the $\beta_i$ are "jointly significant" - either your test outcome as a whole is significant or it isn't. (To be continued...) Commented Jul 2 at 14:09
• (Continuation:) If you use the null hypothesis that all $\beta_i$ are jointly zero, rejecting this means that you have evidence that at least one of your betas is nonzero. This is the negation of the null hypothesis. It will not mean that you have evidence that all betas are nonzero. If you want the latter, you need a test of the null hypothesis that "at least one $\beta_i$ is zero", which is a very different thing. (I'm not quite sure which one of these you want, or whether you want something even different, but I haven't put much time into thinking about your "story".) Commented Jul 2 at 14:12

## 3 Answers

You can do this using a combination of weighted least squares (to account for likely heteroskedasticity), an F-test, and an appropriately structured linear regression.

First, we construct our regression. The target variable will be constructed by stacking the ten individual target variables $$(y_i)$$ into a single vector $$y$$:

$$y = \begin{bmatrix}y_1\\ y_2 \\ \vdots \\ y_{10}\end{bmatrix}$$

The feature matrix is constructed from the original feature vector $$x$$ in the obvious way:

$$X = \begin{bmatrix}x & 0 & \dots & 0\\ 0 & x & \dots & 0 \\&&\vdots \\0 & 0 & \dots &x\end{bmatrix}$$

In this case, we also need to account for the possibility of different intercepts by arranging vectors of ones of appropriate lengths in the same manner as for the $$X$$-matrix. If each industry $$i$$ has $$n_i$$ observations, then the corresponding vector of ones will have length $$n_i$$ and can be denoted $$1_i$$:

$$Z = \begin{bmatrix}1_1 & 0 & \dots & 0\\ 0 & 1_2 & \dots & 0 \\ &&\vdots \\ 0 & 0 & \dots &1_{10}\end{bmatrix}$$ The regression model is then:

$$y = X\beta + Z\gamma + e$$

where the elements of $$\beta$$ correspond to the individual $$\beta_i$$ of the ten (supposedly) unrelated industries.

We can then perform a weighted least squares, estimating the ten weights from the residuals of a preliminary unweighted linear regression. The null hypothesis is that the true model is:

$$y = Z\gamma + e$$

which is nested within the more complex model; as a result, we can use an F-test.

An example in R follows:

# We have 20 observations for each of 10 experiments
x <- rnorm(20)

# Induce heteroskedasticity
y <- rnorm(200) + rnorm(200) * sort(rep(1:10,10)) / 10

# Construct X and Z
X <- kronecker(diag(1,10,10), x)
Z <- kronecker(diag(1,10,10), rep(1,20))

# Weighted least squares - estimating weights
m1 <- lm(y~X+Z-1)
resids <- residuals(m1)
wts <- rep(0,200)
for (i in 1:10) {
start <- 1 + 20*(i-1)
end <- 20*i
wts[start:end] <- 1 / var(resids[start:end])
}

# Weighted least squares
m0 <- lm(y~Z-1, weights = wts)
mw <- lm(y~X+Z-1, weights = wts)

anova(m0, mw)


which in the case of the one run I made generated the rather fortuitous result:

    Analysis of Variance Table

Model 1: y ~ Z - 1
Model 2: y ~ X + Z - 1
Res.Df    RSS Df Sum of Sq      F Pr(>F)
1    190 205.90
2    180 195.83 10    10.073 0.9259 0.5107


Edit in response to comments:

The coefficients of model mw will be the same as if we had run each regression separately. Note that Z is the intercept term, hence the rearrangement of the coefficients in the first line to correspond with how coefficients are typically printed out:

> coef(mw)[c(11, 1 ,12, 2)]
Z1          X1          Z2          X2
0.13063970 -0.08919575  0.52184968  0.57866930

> coef(lm(y[1:20]~x))
(Intercept)           x
0.13063970 -0.08919575

> coef(lm(y[21:40]~x))
(Intercept)           x
0.5218497   0.5786693

• Nice method, jbowman! But a question: are the $\beta$'s in your equation mw equal to the ones you get if you estimate the model separately for each industry? If not, would that not be a problem? And does the omnibus F test, if insignificant, preclude that for one or several industries the $\beta$ differs still from zero significantly?
– BenP
Commented Jul 5 at 18:14
• Good call-out! Yes, they are; I'll add that into the text, with a demo. As a side note, the only thing the heteroskedasticity adjustment achieves, as usual, is give you more accurate standard errors. Commented Jul 5 at 18:18
• Oh thats nice indeed! (+1) Btw, I would not expect the std.errors to be different..Or maybe I should say, if they are different, would that be good? Would it not be desirable that the results are equal to the separate analyses?
– BenP
Commented Jul 5 at 18:20
• @BenP - you are correct, they aren't different!. I was thinking of what the usual effect of heteroskedasticity adjustment is, but not deeply enough about this particular problem! Commented Jul 5 at 18:31
• Could that be obtained by using gls and allowing for different variances?
– BenP
Commented Jul 5 at 18:31

Testing the null hypothesis $$\beta_{1x}=\ldots=\beta_{10x}=0$$ assuming that the standard assumptions hold, two possibilties come immediately to mind. One is that you can test the null hypotheses $$\beta_{ix}=0$$ separately for all $$i$$ and the apply the Bonferroni correction, i.e., you'd reject if the smallest p-value is smaller than $$\alpha/10$$ with significance level $$\alpha$$. This is likely to be quite conservative.

Probably it's better to state the problem as multivariate regression with a single 10-dimensional output vector $$(y_1,\ldots,y_{10})$$. You can then run a likelihood ratio test of the null hypothesis that the $$\beta$$-vector is 0, i.e., $$\beta_{1x}=\ldots=\beta_{10x}=0$$. Unfortunately on the web it's hard to find a good reference for multivariate regression and explanation how to do this software-wise. There is more on MANCOVA, which is in fact more complicated (because on top of your $$x$$ it would have explanatory factors). From the help page of the mancova function in the R-package jmv I'd think that this may run in your situation as well. It also runs some tests and Wilks' Lambda may amount to the Likelihood Ratio test mentioned before, but I'd recommend to have a look at a good book that covers the multivariate linear model to know for sure as I don't have enough time to do the research to explain this exhaustively.

But I thought as long as there's no other answer I tell you what I think. Anybody else is invited to confirm how to test such a standard (I'd think) hypothesis in multivariate regression.

Your goal is to show evidence that the each b-coefficients is zero. This is called "equivalence testing" and often performed by two one-sided tests, abbreviated as "TOST". This can be done in R. I do not have experience with this test procedure, but after some reading I'm convinced that this is much more appropriate in your situation then doing the usual t-tests. Think about it: which significance level should you use? If a t-value does not significantly differ from zero for $$\alpha=0.05$$, this only tells you that the t does not belong to the 5% most extreme t-values if the null hypothesis is true. But it could still be quite extreme e.g. belong to the 20% most extreme t-values under the null, or to the 50% most extreme, etc. And, now that I think about it, taking precautions against to easily rejecting a null hypothesis in case of doing 10 tests (e.g. with Bonferroni etc) would even make the situation worse, in the sense that you would make it even less likely to reject any of your null hypotheses. Actually, the situation should be turned around, the null hypothesis being that the b coefficient differs from zero and the alternative that it doesn't. This is (very roughly) what equivalence tests do, as far as I could see.