does p-value distribution indicate the suitability of the statistical test This post suggests that p-value follows uniform distribution in case of point null hypothesis and continuous data.
In my project, I have millions of p-values from actual data. I don't want to get into the details of my data and statistic test. I just want to know the following:
In general, for point null hypothesis and continuous data, if the p-values do not follow uniform distribution, does it imply that 1) the null hypothesis is not correct, and/or 2)that applied statistic test is not appropriate? could there be other reasons?
Thanks,
 A: P-values may be skewed for reasons that do not imply anything about whether the null hypothesis is correct or not; researchers only do research when the have a good reason to believe their hypothesis is valid. In that situation, we would expect p-values to be skewed high, because bad hypotheses have been filtered out by other means (like common sense, or domain knowledge). In this case the lack of normality is due to skill.
On the other hand, there could be a bias in published results one explanation is that  researchers are more likely to publish p-values that are significant, and put the non-significant ones in a filing cabinet. That would inflate p-values and tend to cast doubt on results. Take this example: If a hundred scientists did the same research project, and only 5% got high p-values, and only those 5% published the results would look to be well established, but in fact just represent random variation.
Overall, I am skeptical if any conclusion can be drawn from the distribution of p-values alone.
A: The distribution of the p-value under the null hypothesis can tell you whether the test achieves its actual size or not. If, say, less than 5% of the distribution falls under the 0.05 mark, the test is said to be anticonservative (rejects too often), and it's conservative for more than 5%. Fisher's Exact Test is a commonly accepted conservative test. P-under-null having a uniform distribution isn't a requirement. It also doesn't make a test "suitable" either - it could have no power, or less power than another test of the same size.
