Justification of one-tailed hypothesis testing

Question

I understand two-tailed hypothesis testing. You have $H_0 : \theta = \theta_0$ (vs. $H_1 = \neg H_0 : \theta \ne \theta_0$). The $p$-value is the probability that $\theta$ generates data at least as extreme as what was observed.

I don't understand one-tailed hypothesis testing. Here, $H_0 : \theta\le\theta_0$ (vs. $H_1 = \neg H_0 : \theta > \theta_0$). The definition of p-value shouldn't have changed from above: it should still be the probability that $\theta$ generates data at least as extreme as what was observed. But we don't know $\theta$, only that it's upper-bounded by $\theta_0$.

So instead, I see texts telling us to assume that $\theta = \theta_0$ (not $\theta \le \theta_0$ as per $H_0$) and calculate the probability that this generates data at least as extreme as what was observed, but only on one end. This seems to have nothing to do with the hypotheses, technically.

Now, I understand that this is frequentist hypothesis testing, and that frequentists place no priors on their $\theta$s. But shouldn't that just mean the hypotheses are then impossible to accept or reject, rather than shoehorning the above calculation into the picture?

Answer 1

That's a thoughtful question. Many texts (perhaps for pedagogical reasons) paper over this issue. What's really going on is that $H_0$ is a composite "hypothesis" in your one-sided situation: it's actually a set of hypotheses, not a single one. It is necessary that for every possible hypothesis in $H_0$, the chance of the test statistic falling in the critical region must be less than or equal to the test size. Moreover, if the test is actually to achieve its nominal size (which is a good thing for achieving high power), then the supremum of these chances (taken over all the null hypotheses) should equal the nominal size. In practice, for simple one-parameter tests of location involving certain "nice" families of distributions, this supremum is attained for the hypothesis with parameter $\theta_0$. Thus, as a practical matter, all computation focuses on this one distribution. But we mustn't forget about the rest of the set $H_0$: that is a crucial distinction between two-sided and one-sided tests (and between "simple" and "composite" tests in general).

This subtly influences the interpretation of results of one-sided tests. When the null is rejected, we can say the evidence points against the true state of nature being any of the distributions in $H_0$. When the null is not rejected, we can only say there exists a distribution in $H_0$ which is "consistent" with the observed data. We are not saying that all distributions in $H_0$ are consistent with the data: far from it! Many of them may yield extremely low likelihoods.

Answer 2

I see the $p$-value as the maximum probability of a type I error. If $\theta \ll \theta_0$, the probability of a type I error rate may be effectively zero, but so be it. When looking at the test from a minimax perspective, an adversary would never draw from deep in the 'interior' of the null hypothesis anyway, and the power should not be affected. For simple situations (the $t$-test, for example) it is possible to construct a test with a guaranteed maximum type I rate, allowing such one sided null hypotheses.

Answer 3

The $p$-value is the probability of the respective event under the condition that $H_0$ is true. The simplest possible toy example are two coin tosses. The 2-sided $H_0$ would be that you consider the coin fair, i.e. you throw one head and one tail. The probability for that is $0.5$. $H_1$ in this case is that you consider it biased to one side or the other, i.e. you either throw two heads or two tails. The probability again is $0.5$

For one 1-sided $H_0$ think of a game where you set your money on heads. You are ok with the coin being fair but of course also comfortable with it being biased towards heads. This is your $H_0$ where you have the possibilities of one head and one tail or two heads: $0.75$ probability. $H_1$ is just the remaining case of two tails where you would call foul: $0.25$ probability. Please note that because you consider the whole region from fair to being biased towards heads as your default two tails is to be considered much more improbable and even more suggestive that something is not in order.

Now when the events of our $H_1$'s happen nevertheless their probabilities are the p-values under the condition that the respective $H_0$'s are true - as noted above. So depending on your confidence level you can or cannot reject your $H_0$'s.

You can experiment with this toy example in R yourself, you should also try different absolute numbers and combinations of heads and tails:

> binom.test(2,2,alternative="two.sided")

    Exact binomial test

data:  2 and 2
number of successes = 2, number of trials = 2, p-value = 0.5
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.1581139 1.0000000
sample estimates:
probability of success 
                     1

> binom.test(2,2,alternative="greater")

    Exact binomial test

data:  2 and 2
number of successes = 2, number of trials = 2, p-value = 0.25
alternative hypothesis: true probability of success is greater than 0.5
95 percent confidence interval:
 0.2236068 1.0000000
sample estimates:
probability of success 
                     1 

Answer 4

You would use a one-sided hypothesis test if only results in one direction are supportive of the conclusion you are trying to reach.

Think of this in terms of the question you are asking. Suppose, for example, you want to see whether obesity leads to increased risk of heart attack. You gather your data, which might consist of 10 obese people and 10 non-obese people. Now let's say that, due to unrecorded confounding factors, poor experimental design, or just plain bad luck, you observe that only 2 of the 10 obese people have heart attacks, compared to 8 of the non-obese people.

Now if you were to conduct a 2-sided hypothesis test on this data, you would conclude that there was a statistically significant association (p ~ 0.02)between obesity and heart attack risk. However, the association would be in the opposite direction to that which you were actually expecting to see, hence the test result would be misleading.

(In real life, an experiment that produced such a counterintuitive result could lead to further questions that are interesting in themselves: for example, the data collection process might need to be improved, or there might be previously-unknown risk factors at work, or maybe conventional wisdom is simply mistaken. But these issues aren't really related to the narrow question of what sort of hypothesis test to use.)