0
$\begingroup$

How is it that a particular test statistic can 1) fail to reject the null hypothesis when the alternative is that the true value is not equal to the null value (two tailed test), yet 2) the same test statistic is sufficient to reject the null hypothesis in favour of the alternative that the true value is likely greater than the null value (a one tailed test)?

Intuitively, it would seem to me that it should be easier to argue that something is simply not equal than to specify the direction of inequality, yet significance testing suggests the opposite is true! How do I rationalise this apparent contradiction?

When considering the example of measuring defect rates, in which we want to know simply whether defect rates have exceeded a threshold level or not, if evidence fails to suggest the defect rate has exceeded the threshold (null) value, I can see that the same evidence may or may not actually suggest the opposite is true (that the true defect rate is actually significantly less than the threshold rate).... but....

I can't understand how of the evidence (test statistic) is sufficient to say with 95% confidence that the threshold has been exceeded (say p=0.04) that we can not also say with 95% confidence that the defect rate is not equal to the threshold value? It is what it is!

Should I think of it as if we are only testing the right half of the distribution, in which case, the critical value termed as "95%" is actually only 90% of the area under consideration? Or perhaps more correctly, that the area of rejection for that tail for a given confidence level is twice as large if one tailed? Is it simply, fundamentally, a different statement to make a one tailed as opposed to two tailed assertion, and we must understand the confidence levels differently as a result?

$\endgroup$
4
$\begingroup$

Q. How is it that a particular test statistic can 1) fail to reject the null hypothesis when the alternative is that the true value is not equal to the null value (two tailed test), yet 2) the same test statistic is sufficient to reject the null hypothesis in favour of the alternative that the true value is likely greater than the null value (a one tailed test)?

A. Because the P-value for a two-tailed test is larger than the P-value for a properly framed one-tailed test. Here is an example using $n = 20$ normal observations.

set.seed(714); x = rnorm(20, 203, 20);  mean(x)
## 208.8698

Two-sided test: We wish to test $H_0: \mu = 200$ vs $H_a: \mu \ne 200.$ The sample mean $\bar X = 208.9$ may seem quite a bit different from $\mu_0 = 200.$ The question is whether the difference is significant at the 5% level. Results of a one-sample, two-sided t test from R statistical software show P-value $0.98 > .05,$ so the difference is not significant and we do not reject $H_0.$ The P-value is $P(|T| > 1.7423) = 0.09762,$ where under $H_0$ the test statistic $T \sim \mathsf{T}(19).$

t.test(x, mu=200)

        One Sample t-test

data:  x
t = 1.7423, df = 19, p-value = 0.09762
alternative hypothesis: true mean is not equal to 200
95 percent confidence interval:
 198.2147 219.5250
sample estimates:
mean of x 
 208.8698

One-sided test: Now change the scenario: Suppose the reason for conducting the experiment was that we suspected the true mean might exceed $200.$ We note that the sample mean is indeed larger than $200.$ So we test $H_0: \mu \le 200$ vs. $H_a: \mu > 200.$ Framing this as a one-tailed test, we are essentially saying that if we had observed $\bar X < 200,$ then we would have said, "Well, that was pointless. Doesn't look as if $\mu > 200.$ What's the next experiment on our list?"

Using the same data as before, the one-sided t test has P-value about $0.049 < .05$ (just barely), and so we have suggestive evidence that the true mean exceeds $200.$ Notice that the P-value is half as large as for the two-sided test. Here, $P(T > 1.7423) = 0.04881,$ where the null distribution of the test statistic is $T \sim \mathsf{T}(19).$

t.test(x, mu=200, alt="greater")

        One Sample t-test

data:  x
t = 1.7423, df = 19, p-value = 0.04881
alternative hypothesis: true mean is greater than 200
95 percent confidence interval:
 200.0671      Inf
sample estimates:
mean of x 
 208.8698 

Q. Intuitively, it would seem to me that it should be easier to argue that something is simply not equal than to specify the direction of inequality, yet significance testing suggests the opposite is true! How do I rationalise this apparent contradiction?

A. Frequentist statisticians like to imagine that they bring no 'prior knowledge' to the analysis of an experiment. There is indeed prior knowledge affecting the entire scenario. Someone had to figure out that $n = 20$ was about the right sample size. (This involves guessing the population SD $\sigma$ and deciding how big a difference is of practical importance.) Someone had to figure out how to do the measurements. (If they're weights, then is measuring to be done on a laboratory balance, a postal scale, or a truck scale?) Also, by doing a t test we're assuming the population is normal. (Only $n = 20$ observations is hardly enough for a conclusive Shapire-Wilk test.)

But even if you don't recognize these factors as 'prior knowledge', it certainly seems clear that stating ahead of time that we believe $\mu > 200$ (if there is any change) does bring relevant information to the table. So if we're doing a one-sided test based on prior opinion about the situation, that information makes it easier to make a decision that we have a significant result. [If we decided to do a one-tailed test only after seeing that $\bar X > 200,$ then we are cheating. It may be easy to rationalize, "I really knew all along: Of course if its different, $\mu$ really has to be bigger," but this is still cheating.]

Finally, you mentioned confidence intervals in your Question: Notice that the lower bound 200.5 of the one-sided 95% CI in the one-sided t procedure, is larger than the lower bound in the two-sided 95% CI $(198.2, 219.5).\,$ In the one-sided procedure we've essentially decided values of $\mu < 200$ aren't of interest. Much of your last paragraph speaks to some of these points.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.