3
$\begingroup$

This post suggests that p-value follows uniform distribution in case of point null hypothesis and continuous data.

In my project, I have millions of p-values from actual data. I don't want to get into the details of my data and statistic test. I just want to know the following:

In general, for point null hypothesis and continuous data, if the p-values do not follow uniform distribution, does it imply that 1) the null hypothesis is not correct, and/or 2)that applied statistic test is not appropriate? could there be other reasons?

Thanks,

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ I don't understand your question. Under these conditions the p-value under the null hypothesis is uniformly distributed by definition, as explained in the comments on & answers to the question you linked to. $\endgroup$
    – Scortchi ♦
    Commented Apr 14, 2016 at 15:22
  • 1
    $\begingroup$ No, it could mean that the null hypothesis is wrong... but to compare the p-value distribution with the uniform distribution, you need to perform the same study dozends of times, which is normally pathetic. $\endgroup$
    – Michael M
    Commented Apr 15, 2016 at 16:13
  • 3
    $\begingroup$ As @MichaelM says, the distribution under an alternative hypothesis won't necessarily be uniform - indeed shouldn't be if it's an alternative you want your test to have power against. I'm also puzzled as to how you're getting the distribution of actual p-values. Perhaps add an illustration of what you mean to your question. $\endgroup$
    – Scortchi ♦
    Commented Apr 15, 2016 at 16:24
  • 4
    $\begingroup$ I want to raise pretty much the same points as @Scortchi. How do you know "if the p-value of a statistic test does not follow uniform distribution"? You should make it clear whether you are talking about running a simulation to generate simulated p-values, or if you intend to use p-values from actual studies. $\endgroup$
    – Silverfish
    Commented Apr 15, 2016 at 18:06
  • 1
    $\begingroup$ Are you obtaining different p values for the same hypothesis or just the same type of hypothesis (but e.g. from different variables)? $\endgroup$
    – Michael M
    Commented Apr 18, 2016 at 14:28

2 Answers 2

Reset to default
1
$\begingroup$

P-values may be skewed for reasons that do not imply anything about whether the null hypothesis is correct or not; researchers only do research when the have a good reason to believe their hypothesis is valid. In that situation, we would expect p-values to be skewed high, because bad hypotheses have been filtered out by other means (like common sense, or domain knowledge). In this case the lack of normality is due to skill.

On the other hand, there could be a bias in published results one explanation is that researchers are more likely to publish p-values that are significant, and put the non-significant ones in a filing cabinet. That would inflate p-values and tend to cast doubt on results. Take this example: If a hundred scientists did the same research project, and only 5% got high p-values, and only those 5% published the results would look to be well established, but in fact just represent random variation.

Overall, I am skeptical if any conclusion can be drawn from the distribution of p-values alone.

Improve this answer
$\endgroup$
1
$\begingroup$

The distribution of the p-value under the null hypothesis can tell you whether the test achieves its actual size or not. If, say, less than 5% of the distribution falls under the 0.05 mark, the test is said to be anticonservative (rejects too often), and it's conservative for more than 5%. Fisher's Exact Test is a commonly accepted conservative test. P-under-null having a uniform distribution isn't a requirement. It also doesn't make a test "suitable" either - it could have no power, or less power than another test of the same size.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.