17
$\begingroup$

I'm hoping someone can help straighten out a point of confusion for me. Say I want to test whether 2 sets of regression coefficients are significantly different from each other, with the following set up:

  • $y_i = \alpha + \beta x_i + \epsilon_i$, with 5 independent variables.
  • 2 groups, with roughly equal sizes $n_1, n_2$ (though this may vary)
  • Thousands of similar regression will be done simultaneously, so some kind of multiple hypothesis correction has to be done.

One approach that was suggested to me is to use a Z-test:

$Z = \frac{b_1 - b_2}{\sqrt(SEb_1^2 + SEb_2^2)}$

Another I've seen suggested on this board is to introduce a dummy variable for grouping and rewrite the model as:

$y_i = \alpha + \beta x_i + \delta(x_ig_i) + \epsilon_i$, where $g$ is the grouping variable, coded as 0, 1.

My question is, how are these two approaches different (e.g. different assumptions made, flexibility)? Is one more appropriate than the other? I suspect this is pretty basic, but any clarification would be greatly appreciated.

$\endgroup$
  • $\begingroup$ I believe the answers and comments to a similar question may provide some of the clarification you seek. $\endgroup$ – whuber Jul 15 '11 at 18:44
  • $\begingroup$ Thank you whuber. I was familiar with that answer. From the discussion below the accepted answer (and your comments there) I was left with the impression that comparing the coefficients of 2 separate fits was not appropriate. Is a z-test applied to the coefficients from the separate fits incorrect or is it that the dummy variable coding is simply easier and provides an equivalent answer? $\endgroup$ – cashoes Jul 15 '11 at 18:58
  • 1
    $\begingroup$ Please see the last paragraph of my reply ("The main limitation..."). The Z-test is valid assuming the $n_i$ are large (otherwise use a t test) and the estimated standard deviations $SEb_i$ are not too different from each other. Neither approach is best when the standard deviations differ a lot (roughly, more than a ratio of 3:1). $\endgroup$ – whuber Jul 15 '11 at 19:14
12
$\begingroup$

The two approaches do differ.

Let the estimated standard errors of the two regressions be $s_1$ and $s_2$. Then, because the combined regression (with all coefficient-dummy interactions) fits the same coefficients, it has the same residuals, whence its standard error can be computed as

$$s = \sqrt{\frac{(n_1-p) s_1^2 + (n_2-p) s_2^2)}{n_1 + n_2 - 2 p}}.$$

The number of parameters $p$ equals $6$ in the example: five slopes and an intercept in each regression.

Let $b_1$ estimate a parameter in one regression, $b_2$ estimate the same parameter in the other regression, and $b$ estimate their difference in the combined regression. Then their standard errors are related by

$$SE(b) = s \sqrt{(SE(b_1)/s_1)^2 + (SE(b_2)/s_2)^2}.$$

If you haven't done the combined regression, but only have statistics for the separate regressions, plug in the preceding equation for $s$. This will be the denominator for the t-test. Evidently it is not the same as the denominator presented in the question.

The assumption made by the combined regression is that the variances of the residuals are essentially the same in both separate regressions. If this is not the case, however, the z-test isn't going to be good, either (unless the sample sizes are large): you would want to use a CABF test or Welch-Satterthwaite t-test.

$\endgroup$
8
$\begingroup$

The most direct way to test for a difference in the coefficient between two groups is to include an interaction term into your regression, which is almost what you describe in your question. The model you would run is the following:

$y_i = \alpha + \beta x_i + \gamma g_i + \delta (x_i \times g_i) + \varepsilon_i$

Note that I have included the group variable as a separate regressor in the model. With this model, a $t$-test with the null hypothesis $H_0: \delta = 0$ is a test of the coefficients being the same between the two groups. To see this, first let $g_i = 0$ in the above model. Then, we get the following equation for group 0:

$y_i = \alpha + \beta x_i + \varepsilon_i$

Now, if $g_i = 1$, then we have:

$y_i = (\alpha + \gamma) + (\beta + \delta) x_i + \varepsilon_i$

Thus, when $\delta$ is 0, then two groups have the same coefficient.

$\endgroup$
  • $\begingroup$ Thanks for correcting the model (I believe my version above simply enforces that the intercept be the same in both groups...). More to the point, would this then be equivalent to the z-test I posted above? $\endgroup$ – cashoes Jul 15 '11 at 19:02
  • $\begingroup$ If one wanted to test whether an effect is different between more than two groups, would an ANOVA comparing the model $y_i = \alpha + \beta x_i + \gamma g_i + \varepsilon_i$ and the one shown in this answer, $y_i = \alpha + \beta x_i + \gamma g_i + \delta (x_i \times g_i) + \varepsilon_i$ be appropriate? $\endgroup$ – miura Nov 15 '13 at 22:43
  • $\begingroup$ @matt-blackwell is this conceptually the same as stratifying the model by each value of g? (ie. b would be the coefficient of x when g=0, and beta+delta when g=1) Although I appreciate that stratifying does not allow statistical comparison. $\endgroup$ – bobmcpop Mar 9 '18 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.