0
$\begingroup$

How is it that a particular test statistic can 1) fail to reject the null hypothesis when the alternative is that the true value is not equal to the null value (two tailed test), yet 2) the same test statistic is sufficient to reject the null hypothesis in favour of the alternative that the true value is likely greater than the null value (a one tailed test)?

Intuitively, it would seem to me that it should be easier to argue that something is simply not equal than to specify the direction of inequality, yet significance testing suggests the opposite is true! How do I rationalise this apparent contradiction?

When considering the example of measuring defect rates, in which we want to know simply whether defect rates have exceeded a threshold level or not, if evidence fails to suggest the defect rate has exceeded the threshold (null) value, I can see that the same evidence may or may not actually suggest the opposite is true (that the true defect rate is actually significantly less than the threshold rate).... but....

I can't understand how of the evidence (test statistic) is sufficient to say with 95% confidence that the threshold has been exceeded (say p=0.04) that we can not also say with 95% confidence that the defect rate is not equal to the threshold value? It is what it is!

Should I think of it as if we are only testing the right half of the distribution, in which case, the critical value termed as "95%" is actually only 90% of the area under consideration? Or perhaps more correctly, that the area of rejection for that tail for a given confidence level is twice as large if one tailed? Is it simply, fundamentally, a different statement to make a one tailed as opposed to two tailed assertion, and we must understand the confidence levels differently as a result?

$\endgroup$
4
$\begingroup$

Q. How is it that a particular test statistic can 1) fail to reject the null hypothesis when the alternative is that the true value is not equal to the null value (two tailed test), yet 2) the same test statistic is sufficient to reject the null hypothesis in favour of the alternative that the true value is likely greater than the null value (a one tailed test)?

A. Because the P-value for a two-tailed test is larger than the P-value for a properly framed one-tailed test. Here is an example using $n = 20$ normal observations.

set.seed(714); x = rnorm(20, 203, 20);  mean(x)
## 208.8698

Two-sided test: We wish to test $H_0: \mu = 200$ vs $H_a: \mu \ne 200.$ The sample mean $\bar X = 208.9$ may seem quite a bit different from $\mu_0 = 200.$ The question is whether the difference is significant at the 5% level. Results of a one-sample, two-sided t test from R statistical software show P-value $0.98 > .05,$ so the difference is not significant and we do not reject $H_0.$ The P-value is $P(|T| > 1.7423) = 0.09762,$ where under $H_0$ the test statistic $T \sim \mathsf{T}(19).$

t.test(x, mu=200)

        One Sample t-test

data:  x
t = 1.7423, df = 19, p-value = 0.09762
alternative hypothesis: true mean is not equal to 200
95 percent confidence interval:
 198.2147 219.5250
sample estimates:
mean of x 
 208.8698

One-sided test: Now change the scenario: Suppose the reason for conducting the experiment was that we suspected the true mean might exceed $200.$ We note that the sample mean is indeed larger than $200.$ So we test $H_0: \mu \le 200$ vs. $H_a: \mu > 200.$ Framing this as a one-tailed test, we are essentially saying that if we had observed $\bar X < 200,$ then we would have said, "Well, that was pointless. Doesn't look as if $\mu > 200.$ What's the next experiment on our list?"

Using the same data as before, the one-sided t test has P-value about $0.049 < .05$ (just barely), and so we have suggestive evidence that the true mean exceeds $200.$ Notice that the P-value is half as large as for the two-sided test. Here, $P(T > 1.7423) = 0.04881,$ where the null distribution of the test statistic is $T \sim \mathsf{T}(19).$

t.test(x, mu=200, alt="greater")

        One Sample t-test

data:  x
t = 1.7423, df = 19, p-value = 0.04881
alternative hypothesis: true mean is greater than 200
95 percent confidence interval:
 200.0671      Inf
sample estimates:
mean of x 
 208.8698 

Q. Intuitively, it would seem to me that it should be easier to argue that something is simply not equal than to specify the direction of inequality, yet significance testing suggests the opposite is true! How do I rationalise this apparent contradiction?

A. Frequentist statisticians like to imagine that they bring no 'prior knowledge' to the analysis of an experiment. There is indeed prior knowledge affecting the entire scenario. Someone had to figure out that $n = 20$ was about the right sample size. (This involves guessing the population SD $\sigma$ and deciding how big a difference is of practical importance.) Someone had to figure out how to do the measurements. (If they're weights, then is measuring to be done on a laboratory balance, a postal scale, or a truck scale?) Also, by doing a t test we're assuming the population is normal. (Only $n = 20$ observations is hardly enough for a conclusive Shapire-Wilk test.)

But even if you don't recognize these factors as 'prior knowledge', it certainly seems clear that stating ahead of time that we believe $\mu > 200$ (if there is any change) does bring relevant information to the table. So if we're doing a one-sided test based on prior opinion about the situation, that information makes it easier to make a decision that we have a significant result. [If we decided to do a one-tailed test only after seeing that $\bar X > 200,$ then we are cheating. It may be easy to rationalize, "I really knew all along: Of course if its different, $\mu$ really has to be bigger," but this is still cheating.]

Finally, you mentioned confidence intervals in your Question: Notice that the lower bound 200.5 of the one-sided 95% CI in the one-sided t procedure, is larger than the lower bound in the two-sided 95% CI $(198.2, 219.5).\,$ In the one-sided procedure we've essentially decided values of $\mu < 200$ aren't of interest. Much of your last paragraph speaks to some of these points.


Addendum based on Comment:

Suppose you have a bent quarter and suspect it may be biased. Not knowing much about bent coins, you test $H_0; p = 1/2$ vs. $H_1: p \ne 1/2$ at the 5% level, where $p$ is the true Heads probability of the quarter. You get $X = 113$ heads in $n = 200$ tosses. The exact P-value of the test is $2P(X \ge 114|p=1/2)=0.077,$ so you cannot reject $H_0$ that the coin is fair. (If you prefer a normal approximation, the z-score is $1.84 < 1.96.)$

2*sum(dbinom(113:200, 200, .5))
[1] 0.07683763

Also, having seen the data, you cannot legitimately claim the coin is biased in favor of Heads because you "really meant" to test $H_0: p \le 1/2$ vs. $H_1: p > 1/2.$ (In this case, you could have rejected: P-value $0.0384 < 0.05$ and $1.84 > 1.645.)$

If you're adamant about bias, you might do another 200 tosses of the coin to see if fresh data support rejection. It's simple to do another 200 tosses and no one is stopping you.

By contrast, now suppose the experiment and test are a for a clinical trial whether a new new drug for treating a virus is effective. If the protocol for the trial calls for testing a two-sided hypothesis, then you can't claim the drug is helpful against the virus.

Worse, you might have trouble getting permission for another clinical trial. Given what some may consider marginal evidence that the drug is useful, many physicians might be reluctant to allow their patients to be randomized to the Control group of a new trial.

This is one reason clinical trials in many countries are subject to approval by supposedly neutral government agencies, that try to make sure the first trial gets it right. The protocol may specify a one-sided test at the 1% level. If successful in this trial, the drug might be subject to additional trials to try to determine for what kinds of patients with what kinds of diagnoses the drug may be considered most effective. (There will probably be continued monitoring of safety and side effects of the drug.)

Comments by people with more extensive recent personal experience than mine, designing various stages of clinical trials, are welcome.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks Bruce. I have to say it still seems odd to me. I guess we have to pick our test before we look at the data, right? I can see where that evens the odds up a bit. But if go for a two tailed test, and we see $1-\alpha <$ effect $<1-\frac{\alpha}{2}$, ok, our test failed ... what stops us then turning around and saying - ah! now we think the effect might be positive! Let's do a one-tailed test ... I guess this "illegal" unless you gather all new data? $\endgroup$ – James Jul 10 at 18:19
  • $\begingroup$ Right about the last part. Not fair to change the 'sidedness' of the test based on results. See Addendum to my Answer. $\endgroup$ – BruceET Jul 10 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.