1
$\begingroup$

This post suggests that p-value follows uniform distribution in case of point null hypothesis and continuous data.

In my project, I have millions of p-values from actual data. I don't want to get into the details of my data and statistic test. I just want to know the following:

In general, for point null hypothesis and continuous data, if the p-values do not follow uniform distribution, does it imply that 1) the null hypothesis is not correct, and/or 2)that applied statistic test is not appropriate? could there be other reasons?

Thanks,

$\endgroup$
  • 3
    $\begingroup$ I don't understand your question. Under these conditions the p-value under the null hypothesis is uniformly distributed by definition, as explained in the comments on & answers to the question you linked to. $\endgroup$ – Scortchi - Reinstate Monica Apr 14 '16 at 15:22
  • 1
    $\begingroup$ No, it could mean that the null hypothesis is wrong... but to compare the p-value distribution with the uniform distribution, you need to perform the same study dozends of times, which is normally pathetic. $\endgroup$ – Michael M Apr 15 '16 at 16:13
  • 3
    $\begingroup$ As @MichaelM says, the distribution under an alternative hypothesis won't necessarily be uniform - indeed shouldn't be if it's an alternative you want your test to have power against. I'm also puzzled as to how you're getting the distribution of actual p-values. Perhaps add an illustration of what you mean to your question. $\endgroup$ – Scortchi - Reinstate Monica Apr 15 '16 at 16:24
  • 3
    $\begingroup$ I want to raise pretty much the same points as @Scortchi. How do you know "if the p-value of a statistic test does not follow uniform distribution"? You should make it clear whether you are talking about running a simulation to generate simulated p-values, or if you intend to use p-values from actual studies. $\endgroup$ – Silverfish Apr 15 '16 at 18:06
  • 1
    $\begingroup$ Are you obtaining different p values for the same hypothesis or just the same type of hypothesis (but e.g. from different variables)? $\endgroup$ – Michael M Apr 18 '16 at 14:28
0
$\begingroup$

P-values may be skewed for reasons that do not imply anything about whether the null hypothesis is correct or not; researchers only do research when the have a good reason to believe their hypothesis is valid. In that situation, we would expect p-values to be skewed high, because bad hypotheses have been filtered out by other means (like common sense, or domain knowledge). In this case the lack of normality is due to skill.

On the other hand, there could be a bias in published results one explanation is that researchers are more likely to publish p-values that are significant, and put the non-significant ones in a filing cabinet. That would inflate p-values and tend to cast doubt on results. Take this example: If a hundred scientists did the same research project, and only 5% got high p-values, and only those 5% published the results would look to be well established, but in fact just represent random variation.

Overall, I am skeptical if any conclusion can be drawn from the distribution of p-values alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.