Return to Question

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $H_1'\colon \mu>0$, then the null becomes $H_0\colon \mu \leq 0$, and so values of $\mu$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $F$-test $H_0\colon \beta=0$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.

Edit 3: Here is a quote from Rob J. Hyndman's blog that, among other things, led to my questions:

Another thing I dislike about statistical tests is the alternative hypothesis. This was not originally part of hypothesis testing as proposed by Fisher. It was introduced by Neyman and Pearson. Frankly, the alternative hypothesis is unnecessary. It is not used in the computation of p-values or for determining statistical significance. The only practical use for the alternative hypothesis that I can see is in determining the power of a test.

(Emphasis is mine.)

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $H_1'\colon \mu>0$, then the null becomes $H_0\colon \mu \leq 0$, and so values of $\mu$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $F$-test $H_0\colon \beta=0$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.

Edit 3: Here is a quote from Rob J. Hyndman's blog that, among other things, led to my questions:

Another thing I dislike about statistical tests is the alternative hypothesis. This was not originally part of hypothesis testing as proposed by Fisher. It was introduced by Neyman and Pearson. Frankly, the alternative hypothesis is unnecessary. It is not used in the computation of p-values or for determining statistical significance. The only practical use for the alternative hypothesis that I can see is in determining the power of a test.

(Emphasis is mine.)

A related question: "Defining extremeness of test statistic and defining $p$-value for a two-sided test".

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $H_1'\colon \mu>0$, then the null becomes $H_0\colon \mu \leq 0$, and so values of $\mu$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $F$-test $H_0\colon \beta=0$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $H_1'\colon \mu>0$, then the null becomes $H_0\colon \mu \leq 0$, and so values of $\mu$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $F$-test $H_0\colon \beta=0$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.

Edit 3: Here is a quote from Rob J. Hyndman's blog that, among other things, led to my questions:

Another thing I dislike about statistical tests is the alternative hypothesis. This was not originally part of hypothesis testing as proposed by Fisher. It was introduced by Neyman and Pearson. Frankly, the alternative hypothesis is unnecessary. It is not used in the computation of p-values or for determining statistical significance. The only practical use for the alternative hypothesis that I can see is in determining the power of a test.

(Emphasis is mine.)

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

So does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

I guess this is how Fisher thought about it, and I am comfortable with it. However, I think I have seen $p$-value being calculated differently in one-sided hypothesis testing. Outcomes that are not in the direction of the alternative do not get considered extreme.

E.g. assume $X\sim N(\mu,\sigma^2)$ and test $$ H_0\colon\mu=0 $$ against $$ H_1\colon\mu\neq 0. $$ Using the empirical mean $\bar x$ as an estimator of $\mu$, the $p$-value is calculated exactly as defined above. If $\bar x$ is far from zero (to either side) in terms of the estimated standard deviation $\hat\sigma$, the $p$-value is low.

Now consider $$ H_1'\colon\mu>0, $$ I have seen $p$-value calculated as $$ \text{p-value}=1-\text{CDF}(t) $$ where $t:=\frac{\bar x}{\hat\sigma/\sqrt{n}}$ is the $t$-statistic and $\text{CDF}$ is the cumulative density function of $t$ under $H_0$. Then $p$-value is high when $\bar x$ is far to the left of zero, contrary to the case above. $\bar x$ being far to the left of zero is extreme in the perspective of $H_0$, but in an uninteresting direction from the perspective of $H_1'$.

Questions: Does the p-value actually depend on the alternative hypothesis? Or is $\text{p-value}=1-\text{CDF}(\bar x)$ nonsense? Or are there alternative definitions depending on whether one uses Fisher's perspective, Neyman-Pearson perspecitve or some mixture of the two?

Edit 1: The definition of the term extreme appears to be crucial. One way of defining extreme is w.r.t. the probability density of the null distribution at the observed result; the lower the density, the more extreme the result. I guess this is how Fisher would have thought (there was a discussion about it somewhere on CV and/or in some paper, I think; I need some time to find it). Another way is to refer to the alternative hypothesis and pick the "interesting" extremes among all, though in my understanding (which could of course be wrong) this would be in conflict with the CV's definition cited above.

Edit 2: Thanks to Alexis for a good catch: if we are to choose an alternative $H_1'\colon \mu>0$, then the null becomes $H_0\colon \mu \leq 0$, and so values of $\mu$ to the left of zero are no longer extreme under the null. So it appears my example was faulty. Let us switch to another example which hopefully illustrates the main point better. In a multiple linear regression model, consider an overall $F$-test $H_0\colon \beta=0$. The alternative is not one-sided, but the distribution of the test statistic under the alternative is to the right of the null distribution, hence only the right tail is "interesting". The questions remain the same.