Would it be accurate to say that a p-value is a random variable whose null distribution is Unif$(0,1)$ which stochastically dominates its distribution under the alternative hypothesis? I realized that I have been thinking about p-values in the sense of "if we set significance level $\alpha$, then we expect $\alpha$ proportion of research papers to be false positives" which I now think is wrong. I'm trying to readjust my idea of what a p-value really is and I haven't seen it phrased the way I put it so I'd like to know if it's correct.
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ I would suggest reading the American Statistical Association's statement on p-values. It discusses a different way to think about p-values that may help you readjust your mindset. amstat.tandfonline.com/doi/full/10.1080/… $\endgroup$– DuncanDec 9, 2016 at 22:36
-
1$\begingroup$ In the discussions that led to the ASA statement on P-values there was a lot of debate about how to include the fact that the P-value is determined by the statistical model as well as the data and the null hypothesis. Little of that made it into the final statement because of concerns about the ability of readers to deal with the notion of a statistical model. Nonetheless it is desirable that the model-based nature of the P-value be kept in mind. $\endgroup$– Michael LewDec 9, 2016 at 22:52
-
$\begingroup$ I like your formulation with stochastic dominance! $\endgroup$– amoebaDec 9, 2016 at 22:54
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Would it be accurate to say that a p-value is a random variable whose null distribution is Unif$(0,1)$ which stochastically dominates its distribution under the alternative hypothesis?
No, that dominance would be a desirable property, not a definition. Many good (and widely used) tests are biased under some alternatives (it's typical for goodness of fit tests for example). Further, one can easily define "useless" test statistics which don't possess this property anywhere under the alternative.
I realized that I have been thinking about p-values in the sense of "if we set significance level $\alpha$, then we expect $\alpha$ proportion of research papers to be false positives" which I now think is wrong.
It is wrong for several reasons. Perhaps the most obvious is that almost all point nulls are actually false (and most published hypothesis tests use point nulls). Consequently while rejections are very common in published papers (in some areas, unfortunately, even ubiquitious) few of them will actually be false rejections even in cases where the hypothesis is not rejected in a later attempt to replicate the result.
$\begingroup$
$\endgroup$
8
$p$ can indeed be regarded as a random variable (in fact, it's a statistic), and is required to be uniformity distributed on $[0, 1]$ under the null hypothesis. However, no guarantees are made about the distribution of $p$ under the alternative hypothesis. This makes sense when you consider that the alternative hypothesis is, for the typical two-tailed test, extremely weak, so very little can be inferred from it.
answered Dec 9, 2016 at 20:43
-
1$\begingroup$ Hmm. Can't one probability distribution be stochastically dominant over another? Without any gambles or choices. $\endgroup$– amoebaDec 9, 2016 at 20:45
-
$\begingroup$ @amoeba I don't see how that would work. How would you define "stochastically dominant"? $\endgroup$ Dec 9, 2016 at 21:40
-
$\begingroup$ $X$ stochastically dominates $Y$ if $F_X(t)\le F_Y(t)$ for all $t$, i.e. if one CDF grows faster than another CDF. I am not an expert though. This is what I read on Wikipedia right now. $\endgroup$– amoebaDec 9, 2016 at 21:43
-
2$\begingroup$ I actually think OP makes a nice observation. For any meaningful test p-vales under H_1 tend to be closer to zero than uniform. Which means that H_0 distribution stoch. dominates the null one. $\endgroup$– amoebaDec 9, 2016 at 22:26
-
1$\begingroup$ A two-tailed one-sample $t$-test tests the null hypothesis $μ = m$, so the alternative hypothesis is $μ ≠ m$, which is an extremely weak statement because it allows $μ$ to have any real value except one, namely $m$. In particular, $μ$ can still be arbitrarily close to $m$. $\endgroup$ Dec 9, 2016 at 22:57
$\begingroup$
$\endgroup$
3
There have been many different definitions of P-values and there will, no doubt, be many more. I do not find yours to be satisfactory because the decision theory-based concept of stochastic dominance seems to fit more with the dichotomous hypothesis test framework than with the significance testing framework that yields a P-value. The hypothesis testing framework utilises rejection regions and the $\alpha$ that you mention in your question, and does not entail calculation of a P-value at all.
See these questions and answers for more on this topic:
What is the difference between "testing of hypothesis" and "test of significance"?
Why are lower p-values not more evidence against the null? Arguments from Johansson 2011
Is it fair to say p-values tell us nothing about the probability null hypotheses are true?
answered Dec 9, 2016 at 23:22
-
$\begingroup$ Maybe the part of my question mentioning hypothesis testing at $\alpha$ didn't accurately address my previous understanding of p-values. And I know that there doesn't have to be an alternative hypothesis or a well-defined distribution under an alternative hypothesis. I guess what I was thinking is that in the case that there is an alternative distribution, or range of distributions, in order to say P is "too small to be uniform(0,1)," we're assuming the P-value under the alternative hypothesis is more concentrated near 0, which was never talked about or proven for tests in my stats classes $\endgroup$– user135912Dec 10, 2016 at 2:05
-
$\begingroup$ Thanks, your answer explains why my description doesn't characterize p-values at all $\endgroup$– user135912Dec 10, 2016 at 2:06
-
$\begingroup$ @Ace You are certainly correct in pointing out the fact that the P-values tend to cluster towards zero when the alternative hypothesis is true or, as I prefer to say, as the true value of the parameter of interest moves away from the null hypothesised value. You might find some ideas of interest in my arXived paper (rejected several times and under slow revision) arxiv.org/abs/1311.0081 $\endgroup$ Dec 10, 2016 at 6:44