3
$\begingroup$

In hypothesis testing, the guidance is to use a one-sided test (alternative "greater" or "lesser") if we don't care about errors in one of the directions. If we do care about errors in both directions and are very good at running one-sided tests, one naïve approach would be to run two one-sided tests for both directions of the alternate hypothesis. For example, in a two-sample t-test for comparing means of two groups, we could run one test for "is group B higher" and another for "is group B lower". How is this approach inferior to running a single two-sided test?

$\endgroup$
2
  • $\begingroup$ To be clear, do you mean two separate tests of the group differences? One "less than" test and one "greater than" test? $\endgroup$ Commented Mar 2, 2020 at 20:25
  • $\begingroup$ One less than and one greater than. $\endgroup$
    – ryu576
    Commented Mar 3, 2020 at 7:46

3 Answers 3

5
$\begingroup$

The combination of one-sided tests that you propose is very close to what a two-sided test actually is, except that the two-sided test has its critical value adjusted to account for the fact that type-1 errors can occur either side of H0.

Lets assume you conduct two one-sided tests, then reject H0 if your test statistic is outside the critical thresholds for either of them.

So if each one-sided test was calibrated such that its type-1 error rate was 5% then the overall type-1 error rate for your combined test would be 10% (assuming the rejection regions for the two sided tests do not overlap). So you need to adjust the threshold for each one-sided test such that it has a type-1 error rate of 2.5%, which then becomes identical to your 2-sided test.

Another way to see the problem is to note that if you use minimum of the p-values from the two one-sided tests as your combined-test p-value, then your combined-test p-value can never be greater than 0.5. Since the p-value under H0 should come from a uniform distribution on [0,1] this test cannot be valid!

If you run this R script a few times you'll see that the smaller p-value from two one-sided t-tests is always half the p-value of the corresponding two-sided test:

y <- rbinom(10,1,0.5)
x <- rnorm(10) + 1*y

p_lower <- t.test(x~y, alternative="less")$p.value
p_higher <- t.test(x~y, alternative="greater")$p.value

p_combined <- min(p_lower, p_higher)

p_twosided <- t.test(x~y, alternative="two.sided")$p.value

p_twosided / p_combined

$\endgroup$
4
  • $\begingroup$ Thank you for the example. If we care about both sides, then the area to the left and the right of the $t$-value doubles the $p$-value. One-sided alternatives make it easier to reject the null hypothesis. Thus, could your last statement also be framed as a $p$-value associated with a one-sided alternative is half that of a $p$-value associated with a two-sided alternative? $\endgroup$ Commented Mar 2, 2020 at 23:28
  • $\begingroup$ Sort of. If you care about both sides then which one-sided alternative would you use? The smaller of the two one-sided p-values is always half the two-sided p-value yes. In this situation one-sided alternatives only make it easier to reject the null because by doing two tests your double your type-1 error rate! $\endgroup$ Commented Mar 2, 2020 at 23:40
  • $\begingroup$ It could be 'two' one-sided tests in the same direction, or 'two' one-sided tests (one for both directions), and the error rate is still 10%. Right? Sorry if I got caught in the weeds with this one. Thank you! $\endgroup$ Commented Mar 3, 2020 at 0:05
  • 1
    $\begingroup$ No it's only 10% because the rejection regions are disjoint. We aren't doing two independent tests with fresh data every time, there is only one dataset so if we did two one sided tests in the same direction we'd still control at 5% because the results would be the same. $\endgroup$ Commented Mar 3, 2020 at 9:56
2
$\begingroup$

In my experience, this approach is not appropriate.

If we do care about errors in both directions and are very good at running one-sided tests, one naïve approach would be to run two one-sided tests for both directions of the alternate hypothesis

This sounds like a less conservative test of the group differences using a two-sided alternative. In keeping with your example, let's say you're interested in testing whether there's a difference in the population means of the two groups. You often want to test the null hypothesis:

$$ H_{0}: \mu_{1} = \mu_{2} $$

against one of three possible alternatives:

\begin{align*} H_{a}: \mu_{1} &> \mu_{2} \\ H_{a}: \mu_{1} &< \mu_{2} \\ H_{a}: \mu_{1} &\neq \mu_{2} \end{align*}

I would advise choosing a two-sided alternative, unless you have a strong theoretical basis to only be interested in one-side. Adopting a two-sided approach effectively splits your $\alpha$ evenly into the two tails. A one-sided alternative assigns $\alpha$ entirely into one of two tails; this is for testing uni-directional effects. As George Savva correctly noted, your combined approach would augment your total error rate to 10% (assuming your $\alpha = .05$).

For example, in a two-sample t-test for comparing means of two groups, we could run one test for "is group B higher" and another for "is group B lower". How is this approach inferior to running a single two-sided test?

Your question is framed in such a way that you are interested in a bi-directional effect. Thus, your alternative should be stated as $H_{a}: \mu_{1} - \mu_{2} \neq 0$ and conducted using one t-test. Note how a two-sided t-test is a more conservative procedure than two sequential one-sided t-tests.

Student's $t$-distribution is simulated below with 8 degrees of freedom. The red vertical lines denote the quantiles (i.e., .025 and .975 quantiles). The area under the curve to the left and to the right indicate the rejection regions under a two-sided alternative. Now, let's assume that our observed test statistic is negative and we default to a one-sided approach. Note, the blue dotted line now denotes the .05 quantile. The area to the left of the given quantile assumes a one-sided alternative; this puts $\alpha$ entirely in the left tail. Thus, less extreme values could yield significant results under this alternative.

enter image description here

Performing a second test, now with $\alpha$ entirely in the right tail, is indicative of two things: (1) it de facto assumes no precedence with respect to directionality; (2) it affords you greater power to detect a significant difference between groups in either direction. Put differently, two 'one-side' tests in both directions is allowing you to ignore a relationship in either direction, while also investigating effects in both directions. It's comical as I think about it.

In sum, this approach is naive because you're being agnostic about the direction of the effect, while also granting yourself more power to detect an effect.

I argue for specificity when stating hypotheses.

$\endgroup$
1
$\begingroup$

Maybe I can hop onto this train. I'm just an engineer and no stats expert, but I would agree with George Sawa. To my understanding, there should be no difference between a two tailed test and two one tailed tests at half the significance level. There is another question that has always been bothering me and which I think is interesting in this context: Why do we need to test for significance on the "other side" of the hypothesized mean in the first place? Why do we lose significance if we test only to the side where the sample mean falls? After all, a sample with a mean greater than the one we suppose gives no hint that the true mean of it`s population is actually smaller.

$\endgroup$
8
  • 3
    $\begingroup$ Welcome to CV. You implicitly assume the two-sided tests are symmetric. Not all are. The issue arises with test statistics that have strongly asymmetric distributions and aren't conveniently transformed into symmetric distributions. This includes the common chi-squared and F ratio statistics. The rest of your post sounds like a set of questions that would be better posed in separate threads, but note that "test only on the side where the sample mean falls" is clearly a hypothesis developed from the data themselves, which makes it invalid. $\endgroup$
    – whuber
    Commented Mar 3, 2020 at 16:38
  • $\begingroup$ Ok, so where is the broken link in this chain of arguments: Every two tailed test can be executed as a one tailed test by splitting the significance level in half, looking at the direction of the sample mean, and then testing in that direction as that is where you would find the significance and the test on the other side will always come back insignificant. Now one could argue that by following this procedure for every drawn sample, your risk of rejecting the null $\mu= \mu_0$ given it's true equals the significance level of the one tailed test and the two tailed test becomes unnecessary. $\endgroup$ Commented Mar 4, 2020 at 8:06
  • $\begingroup$ This sounds like a version of a standard description of the textbook situation of comparing a population mean to a number using a test statistic that is continuously distributed. The conclusion is strange, though: the necessity for a two-tailed test is imposed on us by the null and alternative hypotheses. However you describe it or execute it, it will remain a two-tailed test. $\endgroup$
    – whuber
    Commented Mar 4, 2020 at 14:47
  • $\begingroup$ Gerrymandering the one-sided $p$-value to mimic a two-sided approach is scientifically untenable in my estimation. We shouldn’t let the observed value of the test statistic determine the direction of the test. To address the original question, testing both alternatives implicitly assumes the absence of a theoretical framework for only being interested in one side. Thus, why not default to a two-sided alternative? I agree with whuber that formulating hypotheses from the data is not appropriate. $\endgroup$ Commented Mar 4, 2020 at 19:32
  • $\begingroup$ What I said before indeed is not entirely correct. By following the described procedure one actually would reject the null at twice the significance level of the one sided test (as it is equally likely to hit either tail of the hypothesized distribution), which conforms with the theory. Yet i think we can state there is no difference between a two tailed test and a one tailed test at half the significance level. Another question: Say we tested a drug on a group of people and found it to have a significant effect. Does that actually mean anything? After all many tests are carried out only once. $\endgroup$ Commented Mar 5, 2020 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.