# Does p-value ever depend on the alternative?

Our tag definition of the $$p$$-value says

In frequentist hypothesis testing, the $$p$$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $$p$$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $$X\sim N(\mu,\sigma^2)$$ and test $$H_0\colon\mu=0$$ against $$H_1\colon\mu\neq 0.$$ Using the empirical mean $$\bar x$$ as an estimator of $$\mu$$, the $$p$$-value is calculated exactly as defined above. If $$\bar x$$ is far from zero (to either side) in terms of the estimated standard deviation $$\hat\sigma$$, the $$p$$-value is low.

Now consider $$H_1'\colon\mu>0,$$ I have seen $$p$$-value calculated as $$\text{p-value}=1-\text{CDF}(t)$$ where $$t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$$ is the $$t$$-statistic and $$\text{CDF}$$ is the cumulative density function of $$t$$ under $$H_0$$. Then $$p$$-value is high when $$\bar x$$ is far to the left of zero, contrary to the case above. $$\bar x$$ being far to the left of zero is extreme in the perspective of $$H_0$$, but in an uninteresting direction from the perspective of $$H_1'$$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $$\text{p-value}=1-\text{CDF}(\bar x)$$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $$H_1'\colon \mu>0$$, then the null becomes $$H_0\colon \mu \leq 0$$, and so values of $$\mu$$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $$F$$-test $$H_0\colon \beta=0$$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.

Edit 3: Here is a quote from Rob J. Hyndman's blog that, among other things, led to my questions:

Another thing I dislike about statistical tests is the alternative hypothesis. This was not originally part of hypothesis testing as proposed by Fisher. It was introduced by Neyman and Pearson. Frankly, the alternative hypothesis is unnecessary. It is not used in the computation of p-values or for determining statistical significance. The only practical use for the alternative hypothesis that I can see is in determining the power of a test.

(Emphasis is mine.)

• Another example could probably be $F$-test of joint significance of several regressors in a multiple linear regression. The alternative is not one-sided, but given sufficient estimation accuracy, the test statistic will be in the right tail of the distribution under the alternative (not in the left tail), so only the right tail is "interesting". – Richard Hardy Feb 2 '20 at 14:26
• The alternative tells you which direction(s) more extreme is in. – Glen_b Feb 2 '20 at 15:11
• For all frequentist hypothesis tests, the alternative hypothesis is what the test provides evidence for (upon rejection) or fails to find evidence for (failure to reject). Also note that in $H_{0}: \mu \le \mu_{0}$ with $H_{1}: \mu > \mu_{0}$ the p value "depends upon the alternative" in that $H_{1}$ is the complement of $H_{0}$ (so $H_{1}$ tells you everything about $H_{0}$, and vice versa. This complemetarity is also true when the null takes the form $H_{0}: \mu = \mu _{0}$, $H_{0}: \mu \ge \mu _{0}$, or $H_{0}: |\mu - \mu_{0}| \ge \varepsilon$. – Alexis Feb 2 '20 at 18:01
• "More extreme" is usually not defined or is mischaracterized. That's why I made the effort to explain this at stats.stackexchange.com/a/130772/919. As far as I can tell, that post addresses all the questions you have posed here. – whuber Feb 2 '20 at 18:54
• @RichardHardy given the title literally assumed an alternative, I was explaining how it works in that specific situation, not giving a lengthy exposition that covers all the potential cases not mentioned by it. – Glen_b Feb 2 '20 at 22:20

The test statistic ($$t$$ in your example) and all calculations to reach that point depend only on the null hypothesis $$H_0$$ and nothing else.

The p-value is affected by the alternative hypothesis $$H_1$$ as the $$H_1$$ identifies which values are considered as "extreme" values and the p-value calculates the proximity of the final result (your $$t$$) to those values.

For instance, in your example of $$H_0$$ vs $$H_1'$$ you would reject $$H_0$$ only if $$t>T_\alpha$$ and for the example $$H_0$$ vs $$H_1$$ you would reject $$H_0$$ only if $$t>T_{\alpha/2}$$ or $$t<-T_{\alpha/2}$$.

Thus the p-value of $$H_0$$ vs $$H_1$$ would be the probability of the union of two sets whereas the $$H_0$$ vs $$H_1'$$ would be the probability of one set where the cut-off point is higher on the x-axis compared to the previous case.

EDIT: In response to what you mentioned about Fisher, I believe you are referring to the famous lady testing test. Which indeed doesn't have a strictly speaking alternative hypothesis but it is slightly different compared to the hypothesis tests that we usually conduct.

In this example, he only defined the null hypothesis $$H_0$$: She has no ability to distinguish the tea and he used the combination formula to measure the probability of all possible outcomes given that $$H_0$$ is true which is essentially the p-value of each data point.

The main difference/trick here that you might be looking for is that in Fisher's eyes, it would only take one incorrect guess to make her a liar and thus he wanted to identify the smallest amount of cups that he needs to give her to taste. In a sense, one might say that he tested the $$H_1:$$At least one incorrect guess and he looked for the smallest possible sample size for some pre-defined parameters.

This is a slightly different case to the way we usually conduct statistical hypothesis tests as we take sample from a population and we usually "allow" some non-$$H_0$$ cases. I guess the final answer to your question is that we want an $$H_1$$ or at least a "loose-definition" of it in order to define what are the "extreme-departures" from the $$H_0$$ (Even if you are Fisher and you hide it well enough).

Really good question by the way :)

• Thank you for your answer! It is bridging the gap pretty nicely. However, I think Fisher did not consider an alternative hypothesis (it came later with Neyman-Pearson), yet he defined the $p$-value. Could you please comment on that? – Richard Hardy Feb 2 '20 at 19:29
• Ah yes, I think you know to what you are referring to, let me edit my answer – Vasilis Vasileiou Feb 2 '20 at 19:52
• Thanks for the update. I think I found what I was looking for, thanks to @whuber's comment and Spanos "Probability Theory and Statistical Inference" (1999), Chapter 14. The latter chapter contains what I must have meant. According to my understanding of Spanos interpretation of Fisher, there is no alternative there, while the idea of the test is to see how well the data is compatible with $H_0$. ctd... – Richard Hardy Feb 2 '20 at 21:06
• ...Fisher's implicit alternative is much broader than the explicit alternative of Neyman-Pearson (NP). Under the alternative Fisher allows for the model to be misspecified in an arbitrary way, while NP remain within the specified model. It is too late for me to continue today, but I will return. +1 anyway. – Richard Hardy Feb 2 '20 at 21:08
• @RichardHardy I believe the goal is to test whether the data is incompatible with $H_0$. Fisher's hypothesis testing is a search for anomalies (with regard to the theory that there are no effects). – Sextus Empiricus Feb 2 '20 at 21:14

The reason that results in the wrong direction don’t give small p-values is because they provide terrible evidence in favor of the alternative. Imagine a null hypothesis of a fair coin and an alternative of a bias towards heads. You then flip the coin 100 times and get 99 tails. You have terrible evidence in favor of your alternative hypothesis.

This can apply in other settings. Think of an F-test comparing the variances of two distributions. If you think the distribution with its variance on top has higher variance but wind up with a variance ratio $$<1$$, you have rather poor evidence that the distribution on top has higher variance than the distribution on the bottom.

$$F_0=s_1^2/s_2^2$$

If $$s_1^2, your evidence is quite poor that $$\sigma_1^2>\sigma_2^2$$.

• Thank you for your insight! This is a great answer to a question that I did not ask. But what about the questions that I did ask? – Richard Hardy Feb 2 '20 at 16:03
• I think it does answer your question. When you do one-sided testing in one direction, the p-value is the $1-CDF(\bar{x})$ that you posted. In the other direction, the p-value is just $CDF(\bar{x})$. I’ll make an edit with some pictures later today or tomorrow. – Dave Feb 2 '20 at 16:24
• Thank you, Dave. It is not that I do not understand your example. I also think I got the intuition. My questions are about the definition(s) of $p$-value and appropriate use of terminology. – Richard Hardy Feb 2 '20 at 17:24