# Effective size of two 1-sided tests compared to a 2-sided test

Suppose we are interested in testing the usual hypothesis concerning the value of some parameter that arises from a model: $$\mathcal{H}_0\!: \mu = 0 \\[7pt] {\rm vs} \\[5pt] \mathcal{H}_1\!: \mu \ne 0$$ This test is fixed at a usual size of $$\alpha$$. Suppose we are truly agnostic about the suspected direction of effect, such as with a chemotherapy drug that is effective but very toxic. As such, we want to actually report the direction of effect, though the test may be underpowered to provide reasonable estimates of effect.

I suspect it is inappropriate to report the direction of effect along with the accompanying $$p$$-value from the two-tailed test. The test merely concerns whether the estimate is non-zero. So the correct way of reporting the result is to say, "We reject the null hypothesis, there is evidence to conclude the effect is statistically significantly different from 0." But this is an underwhelming result.

Rather, it seems one should perform two one-sided tests with some attempt to control for multiplicity. The obvious approach would be: first perform test of harm $$\mu > 0$$, then perform test of effectiveness $$\mu < 0$$. Doing two level $$\alpha/2$$ tests would be recommended by Bonferroni—the inefficient procedure for multiple comparisons. However, this yields the same critical values as with the single level 0.05 test, and now it seems I can claim something about the direction of effect.

Did we get something from nothing? Is there a flaw with this line of reasoning?

• I don't follow the reasoning behind "So we would be reporting something not supported by the data." In your narrative the data have led to two conclusions: (1) there's a significant difference and (2) the difference is estimated to be such-and-such. No principle of logic, statistics, or mathematics then prevents you from drawing further conclusions from the estimated difference, including conclusions about its sign.
– whuber
Commented Dec 2, 2019 at 17:21
• @whuber I guess I want to draw the distinction about making claims that are directly supported by the data rather than as a course of ancillary reporting and follow-up. For instance, we understand that the two tailed and one tailed hypotheses are fundamentally different for that reason. Commented Dec 2, 2019 at 17:25
• If your original intention is to perform the two-sided test as well as the estimated difference, it's hard to see any real distinction between "directly supported by the data" and "ancillary reporting and follow-up."
– whuber
Commented Dec 2, 2019 at 17:53

## 1 Answer

I think I follow what you're saying and have produced an R simulation.

set.seed(2019)
x <- rnorm(50,0,1)
y <- rnorm(50,0.5,1)
t0 <- t.test(x,y,alternative="two.sided",var.equal=T)
t1 <- t.test(x,y,alternative="greater",var.equal=T)
t2 <- t.test(x,y,alternative="less",var.equal=T)
p0 <- t0$$p.value p1 <- t1$$p.value
p2 <- t2\$p.value
round(p0,15) == round((1-p1)+p2,15)


My p0 is the p-value from the two-sided test; 1-p1 is the complement of the p-value of the $$\mu_X > \mu_Y$$ test; and p2 is the p-value of the $$\mu_X < \mu_Y$$ test.

Up to 15 decimal places, the two methods give the same information, which tells me that they are the same. Consequently, I say that we get no additional information (nor any wrong information) by doing the two tests, rather than going a two-sided test and inferring effect direction from the sign of the test statistic.

If we have $$\bar{x} - \bar{y} = 3\sigma$$, that is strong evidence against $$\bar{x} = \bar{y}$$ but even stronger evidence against $$\bar{x} < \bar{y}$$.

• I get your point: the t-tests are completely complementary and so holds the rare and unlikely condition that Bonferroni can be efficient. Running more than 1 sims doesn't guarantee 15 decimals of accuracy, though, but the errors are numerical. What's more damning is considering Fisher's Exact Test in moderate/small samples where p1+p2 > 1. Commented Dec 2, 2019 at 20:13