Two one-sided hypothesis tests instead of a two-sided test?

Question

In hypothesis testing, the guidance is to use a one-sided test (alternative "greater" or "lesser") if we don't care about errors in one of the directions. If we do care about errors in both directions and are very good at running one-sided tests, one naïve approach would be to run two one-sided tests for both directions of the alternate hypothesis. For example, in a two-sample t-test for comparing means of two groups, we could run one test for "is group B higher" and another for "is group B lower". How is this approach inferior to running a single two-sided test?

Answer 1

The combination of one-sided tests that you propose is very close to what a two-sided test actually is, except that the two-sided test has its critical value adjusted to account for the fact that type-1 errors can occur either side of H0.

Lets assume you conduct two one-sided tests, then reject H0 if your test statistic is outside the critical thresholds for either of them.

So if each one-sided test was calibrated such that its type-1 error rate was 5% then the overall type-1 error rate for your combined test would be 10% (assuming the rejection regions for the two sided tests do not overlap). So you need to adjust the threshold for each one-sided test such that it has a type-1 error rate of 2.5%, which then becomes identical to your 2-sided test.

Another way to see the problem is to note that if you use minimum of the p-values from the two one-sided tests as your combined-test p-value, then your combined-test p-value can never be greater than 0.5. Since the p-value under H0 should come from a uniform distribution on [0,1] this test cannot be valid!

If you run this R script a few times you'll see that the smaller p-value from two one-sided t-tests is always half the p-value of the corresponding two-sided test:

y <- rbinom(10,1,0.5)
x <- rnorm(10) + 1*y

p_lower <- t.test(x~y, alternative="less")$p.value
p_higher <- t.test(x~y, alternative="greater")$p.value

p_combined <- min(p_lower, p_higher)

p_twosided <- t.test(x~y, alternative="two.sided")$p.value

p_twosided / p_combined

Answer 2

In my experience, this approach is not appropriate.

If we do care about errors in both directions and are very good at running one-sided tests, one naïve approach would be to run two one-sided tests for both directions of the alternate hypothesis

This sounds like a less conservative test of the group differences using a two-sided alternative. In keeping with your example, let's say you're interested in testing whether there's a difference in the population means of the two groups. You often want to test the null hypothesis:

$$ H_{0}: \mu_{1} = \mu_{2} $$

against one of three possible alternatives:

\begin{align*} H_{a}: \mu_{1} &> \mu_{2} \\ H_{a}: \mu_{1} &< \mu_{2} \\ H_{a}: \mu_{1} &\neq \mu_{2} \end{align*}

I would advise choosing a two-sided alternative, unless you have a strong theoretical basis to only be interested in one-side. Adopting a two-sided approach effectively splits your $\alpha$ evenly into the two tails. A one-sided alternative assigns $\alpha$ entirely into one of two tails; this is for testing uni-directional effects. As George Savva correctly noted, your combined approach would augment your total error rate to 10% (assuming your $\alpha = .05$).

For example, in a two-sample t-test for comparing means of two groups, we could run one test for "is group B higher" and another for "is group B lower". How is this approach inferior to running a single two-sided test?

Your question is framed in such a way that you are interested in a bi-directional effect. Thus, your alternative should be stated as $H_{a}: \mu_{1} - \mu_{2} \neq 0$ and conducted using one t-test. Note how a two-sided t-test is a more conservative procedure than two sequential one-sided t-tests.

Student's $t$-distribution is simulated below with 8 degrees of freedom. The red vertical lines denote the quantiles (i.e., .025 and .975 quantiles). The area under the curve to the left and to the right indicate the rejection regions under a two-sided alternative. Now, let's assume that our observed test statistic is negative and we default to a one-sided approach. Note, the blue dotted line now denotes the .05 quantile. The area to the left of the given quantile assumes a one-sided alternative; this puts $\alpha$ entirely in the left tail. Thus, less extreme values could yield significant results under this alternative.

Performing a second test, now with $\alpha$ entirely in the right tail, is indicative of two things: (1) it de facto assumes no precedence with respect to directionality; (2) it affords you greater power to detect a significant difference between groups in either direction. Put differently, two 'one-side' tests in both directions is allowing you to ignore a relationship in either direction, while also investigating effects in both directions. It's comical as I think about it.

In sum, this approach is naive because you're being agnostic about the direction of the effect, while also granting yourself more power to detect an effect.

I argue for specificity when stating hypotheses.

Answer 3

Maybe I can hop onto this train. I'm just an engineer and no stats expert, but I would agree with George Sawa. To my understanding, there should be no difference between a two tailed test and two one tailed tests at half the significance level. There is another question that has always been bothering me and which I think is interesting in this context: Why do we need to test for significance on the "other side" of the hypothesized mean in the first place? Why do we lose significance if we test only to the side where the sample mean falls? After all, a sample with a mean greater than the one we suppose gives no hint that the true mean of it`s population is actually smaller.