I have a regression, e.g. $Y=a+b_1X_1+b_2X_2+c_1Z_1+c_2Z_2+e$.

Is it possible test the one-tailed null $b_1+b_2\leqslant c_1+c_2$ against the alternative $b_1+b_2>c_1+c_2$? Notice that this is an $F$-test with a single restriction.


Yes, this is possible.

The general multiple linear regression model is

$$y = X\beta + \varepsilon$$

with the $k$ regressor values, along with a constant vector, forming the columns of $X$ and the $p=k+1$ coefficients arranged correspondingly in the vector $\beta$. When there is no collinearity among those columns, the least-squares solution is

$$\hat \beta = (X^\prime X)^{-1}X^\prime y.$$

It fits (or "predicts") the expectations $E[y|X]$ with

$$\hat y = X\hat\beta = X(X^\prime X)^{-1}X^\prime y.$$

Assuming the "errors" $\varepsilon$ are uncorrelated and have common variance $\sigma^2$, the unbiased estimator of $\sigma^2$ is

$$\hat\sigma^2 = \frac{||y - \hat y||^2}{n-p}.$$

(This is the mean squared residual $||y - \hat y||^2/n$ adjusted by the factor $n/(n-p)$ to make the estimator unbiased.) The covariance matrix of the estimates $\hat\beta$ can then be estimated as

$$\operatorname{Cov}(\hat\beta) \approx \hat\sigma^2 (X^\prime X)^{-1}.$$

Any linear combination $c^\prime \beta$ will be estimated with the same linear combination of parameter estimates, $c^\prime \hat\beta$. To compare $c^\prime\beta$ to some predetermined constant $c_0$ (often $0$), compute the variance of the estimate as

$$\operatorname{se}(c^\prime\hat\beta) = \operatorname{Var}(c^\prime\hat\beta) = c^\prime \operatorname{Cov}(\hat\beta)c = \hat\sigma^2 c^\prime(X^\prime X)^{-1}c$$

and conduct a t-test (either one- or two-sided) based on the t statistic

$$t = \frac{c^\prime\hat\beta - c_0}{\sqrt{\operatorname{se}(c^\prime\hat\beta) }}.$$

For $n$ observations, it has $n-p$ degrees of freedom.

The following R code shows how to perform these calculations and runs a (fast) simulation demonstrating that one-sided p-values do indeed have a uniform distribution when the null hypothesis holds. The required information is extracted from an lm fit using coef to obtain $\hat \beta$ and vcov for $\hat\sigma^2 (X^\prime X)^{-1}$.

To check the simulation, it conducts a $\chi^2$ test of uniformity; the output for this particular set of 1000 iterations is a p-value of $0.4428$, evidence that the code is performing as claimed.

beta <- c(1,4,2,3)       # The coefficients
n <- 20                  # The number of observations
combo <- c(0,1,1,-1,-1)  # The contrast (starting with an intercept coefficient)
sigma <- 2               # The error SD
n.sim <- 1e3             # Number of iterations of the simulation
# Prepare to simulate.
k <- length(beta)
combo.name <- paste0("Contrast(", paste(combo, collapse=","), ")")
# Simulate with many different sets of regressors and responses, but all
# using the same `beta`.
p.values <- replicate(n.sim, {
  # Create regressors.
  x <- matrix(rnorm(n*k), n)
  colnames(x) <- paste0("X", 1:k)
  y <- x %*% beta + rnorm(n, 0, sigma)
  # Test the combination.
  fit <- lm(y ~ ., as.data.frame(x))
  beta.hat <- crossprod(coef(fit), combo)
  beta.hat.se <- sqrt(combo %*% vcov(fit) %*% combo)
  t.stat <- beta.hat / beta.hat.se
  p <- pt(t.stat, n-k-1) # A one-sided p-value
  # When running this "for real," uncomment the following lines to
  # see the results.
#   stats <- rbind(coef(summary(fit)), c(beta.hat, beta.hat.se, t.stat, p))
#   rownames(stats)[k+2] <- combo.name
#   print(stats, digits=3)
  # Return the p-value for the (two-sided) t-test
# Display the simulation results.
n.bins <- floor(n.sim / 100)
(p.dist <- chisq.test(table(floor(n.bins*p.values)))$p.value) #$
hist(p.values, freq=FALSE, breaks=n.bins,
     sub=paste("p (uniform) =", format(p.dist, digits=3)))
abline(h = 1, col="Gray", lwd=2, lty=3)

Yes this is indeed very possible, in fact nothing changes. Use the standard F-test provided by whatever software you are using.

Actually, when you only have one parameter the F-test is just the square of a the same t-test. That is, $t^2_{n-k-1} \sim F_{1,n-k-1}$. As Whuber points out in the comments, you have to be carefull in thinking about 1 and 2 sided alternatives. And in this case, (for the above results to be true) you need a two sided alternative. But this should always be the case, unless you have a really good argument for why the statistic cannot be negative (or positive).

  • $\begingroup$ So I can just compute the F-stat, take the square root, and see if it is larger than my one-sided critical t-value, e.g. 1.28 for the 10% significance level? $\endgroup$
    – user93929
    Nov 6 '15 at 14:01
  • 1
    $\begingroup$ You seem to be claiming that the one-tailed and two-tailed tests are the same, but obviously they are not. $\endgroup$
    – whuber
    Nov 6 '15 at 14:04
  • $\begingroup$ @whuber, yes indeed. Have updated! $\endgroup$
    – Repmat
    Nov 6 '15 at 14:29
  • $\begingroup$ Is it that simple? It's not obvious to me that this should work... $\endgroup$ Nov 6 '15 at 14:38
  • $\begingroup$ So, I can only test a two-sided alternative and not a one-sided alternative with the F-test with one restriction? $\endgroup$
    – user93929
    Nov 6 '15 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.