0
$\begingroup$

This question is a follow-up to Testing equality of coefficients from two different regressions.

Consider the two data generating processes $$y_1=x_1'\beta_1+e_1$$ and $$y_2=x'_2\beta_2+e_2$$ where $x_1$ and $x_2$ are vectors of the same length. Assume that $x_j$ is independent of $e_j$ for $j=1,2$, and that we have two independent iid samples of sizes $n_1$ and $n_2$ from the first and second data generating process, respectively. Assume $n_1>n_2$ (or $n_1\neq n_2$). For us to be able to use asymptotic theory for least squares I also assume that $E(y^4)<\infty$, $E||x_j||^4<\infty$ for $j=1,2$, and that $E(x_jx_j')$ is positive definite for $j=1,2$.

Is there a test statistic whose asymptotic distribution is known under $H_0:\beta_1=\beta_2$ as $n_1$ and $n_2$ diverges to infinity? I would like to use such a statistic to acquire an asymptotically valid test for $H_0$ against not-$H_0$. One idea is to consider a test statistic based on the first $\min(n_1,n_2)$ observations (see comments).

I believe the test statistics proposed in Testing equality of coefficients from two different regressions do not have known asymptotic distributions.

I tried to use multivariate regression and SUR to create a test statistic, but could not derive relevant asymptotic results once $n_1\neq n_2$.

Improve this question
$\endgroup$
  • $\begingroup$ Are you assuming knowledge of the distribution of the $e_i$? $\endgroup$ – jbowman Feb 26 '20 at 19:18
  • $\begingroup$ @jbowman No. But I am definitely interested in the case where they are normally distributed. $\endgroup$ – Elias Feb 26 '20 at 19:20
  • $\begingroup$ Are you assuming $x_1$ and $x_2$ have the same distribution? Why do we need to assume $n_1\gt n_2$? That looks superfluous. Since you're interested in asymptotics and both $n_i$ are simultaneously diverging, why not develop a statistic from the first $\min(n_1,n_2)$ data in each dataset, thereby reducing your question to the case $n_1=n_2$ for which you imply you have a solution? $\endgroup$ – whuber♦ Feb 26 '20 at 20:32
  • $\begingroup$ @whuber No. For example, their means may be different. I want to assume $n_1\neq n_2$ since my idea was to look at these two regressions as regressions in two different populations of different sizes. Without loss of generality we can look at $n_1>n_2$. Would it not be unwise to through away observations? Suppose $n_1=10$ and $n_2=100$. Then $\min(n_1,n_2)=10$. $\endgroup$ – Elias Feb 26 '20 at 20:35
  • 1
    $\begingroup$ @whuber I understand and agree with your argument under the assumption that we look at the first $\min(n_1,n_2)$ observations and consider $\min(n_1,n_2)\to\infty$, but otherwise I do not see why the difference is asymptotically standard normal (and I believe it is not), for then it has to depend on how we take the limits of $n_1$ and $n_2$. In any case, I see your point. $\endgroup$ – Elias Feb 26 '20 at 20:57
2
$\begingroup$

We can construct the regression:

$$\begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} x_1 & 0 \\ x_2 & x_2 \end{bmatrix} \begin{bmatrix} \beta_1 \\ \Delta \end{bmatrix} + \begin{bmatrix} e_1 \\ e_2 \end{bmatrix}$$

When estimating the parameters, we should keep in mind that $\sigma^2_1$ need not equal $\sigma^2_2$, which introduces a simple, well-structured, heteroskedasticity into the estimation, which can be addressed via weighted least squares.

The null hypothesis is $H_0: \Delta = 0$, and the alternative is, evidently, $H_A: \Delta \neq 0$. In the case of Gaussian errors, the obvious test is an $F$-test, which would be exact in finite samples if it were not for the likely mild) disruption caused by having to estimate two variance terms instead of one.

To see how much of a disruption the heteroskedasticity causes to the distribution of the $F$-statistic, we construct an example with $n_1 = n_2 = 100$ and four regressors in both $x_1$ and $x_2$. We set $\sigma^2_2 = 4\sigma^2_1$, and estimate the regression using iteratively reweighted least squares. We then calculate the p-value of the $F$-test, which, under $H_0$, should be distributed according to a Uniform distribution. Repeating the entire process 10,000 times allows us to test the distribution of the 10,000 p-values against the Uniform distribution, in this case using a Kolmogorov-Smirnov test:

p_values <- rep(0,10000)

Zeroes <- matrix(0, nrow=100, ncol=4)
for (j in 1:length(p_values)) {

  X1 <- matrix(rnorm(400), nrow=100)
  X2 <- matrix(rnorm(400), nrow=100)
  y1 <- rowSums(X1) + rnorm(100)
  y2 <- rowSums(X2) + 2*rnorm(100)

  y <- c(y1, y2)
  X <- rbind(cbind(X1, Zeroes),
             cbind(X2, X2))

  # Iteratively reweighted least squares
  repeat {
    ehat <- residuals(lm(y~X))
    var_ratio <- var(ehat[101:200]) / var(ehat[1:100])
    y[101:200] <- y[101:200] / sqrt(var_ratio)
    X[101:200,] <- X[101:200,] / sqrt(var_ratio)

    if (abs(var_ratio-1) < 0.00001) break
  } 

  m1 <- lm(y~X)
  m2 <- lm(y~X[1:200,1:4])
  p_values[j] <-anova(m1, m2)$`Pr(>F)`[2]
}

ks.test(p_values, "punif")

    One-sample Kolmogorov-Smirnov test

data:  p_values
D = 0.008737, p-value = 0.43
alternative hypothesis: two-sided

which indicates, at least in this case, that the $F$-statistic's distribution is pretty close to the theoretical distribution, even for a not-terribly-large sample size.

In the case of non-Gaussian errors, it seems not unlikely that the $F$-test will break down, due to its known sensitivity to exactly this situation. In that case, an alternative would be to construct a permutation or bootstrap test (https://en.wikipedia.org/wiki/Resampling_(statistics)) based on the $F$-statistic, but I should point out that if you were to do so, there would likely be no particular advantage to sticking with the $F$-statistic as the test statistic of choice. An asymptotically equivalent permutation test can be applied if $n_1$ and $n_2$ together are too large for an exact permutation test; the asymptotics are such that the proper significance level is obtained (asymptotically) as long as exchangeability assumptions are met (and of course the null hypothesis is true.)

You can also rely on the asymptotic distribution of the $F$-statistic under non-Gaussianity, as outlined here: Does the F-test for multivariable regression work with non-normal residuals but large sample size?. However, IIRC the power of the $F$-test can be severely affected even if its significance level is approximately correct, hence my recommendation for the use of a permutation-based test.

Improve this answer
$\endgroup$
  • $\begingroup$ What about the asymptotic distribution of the $F$-statistic? You write "should be distributed as...". Why not go with whuber's suggestion? $\endgroup$ – Elias Feb 26 '20 at 20:52
  • $\begingroup$ If it has the $F$ distribution in a finite sample, it's going to have it asymptotically too. Who cares about asymptotics if you know the finite sample properties well enough? $\endgroup$ – jbowman Feb 26 '20 at 20:54
  • $\begingroup$ Sorry. I meant to ask about the asymptotic distribution of the $F$-statistic under non-Gaussian errors. $\endgroup$ – Elias Feb 26 '20 at 20:55
  • $\begingroup$ Also note that the $F$-statistic is equivalent to the $t$-statistic if you only have one coefficient you are testing! $\endgroup$ – jbowman Feb 26 '20 at 20:55
  • 1
    $\begingroup$ I've added a paragraph to address your question. $\endgroup$ – jbowman Feb 26 '20 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.