Let's say I performed 100 tests and and want to correct for multiple comparisons. Before I do so, I plot the unaltered p values in a histogram to see what the distribution looks like.

If the null were true, you would expect that a histogram of unaltered p values would be a uniform distribution. If it was right skewed, you might conclude that overall, there seems to be something real going on.

However, what if you got something like:

enter image description here

What does this mean?

  • 3
    $\begingroup$ Based on the count values on your graph I would say you have a very small sample hence why this unusual result. You do not have enough p-values to really check how they are distributed (there is too much variation). $\endgroup$ – user2974951 Dec 21 '18 at 11:48
  • $\begingroup$ Right you are correct that the sample size is small (n = 36), but despite that, I am still curious what it would mean if say n = 1000 $\endgroup$ – hlinee Dec 21 '18 at 12:24
  • $\begingroup$ The interpretation would be similar to when you have right or left skewed p-values. When you have most of your p-values on the left side you would conclude that in most cases we can find an effect, if most of your p-values were on the right side you would conclude that in most cases we cannot find an effect. If most p-values were in the middle you would conclude that in most cases we cannot find an effect but we are not too sure in this (for ex. there is still 50 % chance of this outcome happening). $\endgroup$ – user2974951 Dec 21 '18 at 12:31

If you performed N independent tests, any number of which might be probing real or null effects, there is no telling exactly what distribution of p-values you ought to expect.

As you said, if the null was true in all cases, then you would expect a uniform distribution of p-values across your N tests (bearing in mind that in a small sample, the histogram of p-values may show large variability around the population distribution). This is a special case where it is easy to derive the expected distribution of p-values.

If you knew exactly the true population statistics for each test you were doing, then it might also be possible to derive mathematically the expected distribution of p-values for each test, and your overall distribution of p-values across tests would then be a mixture of those distributions. This seems rather complicated to do, but in principle I suppose you could work it out, if you knew the necessary information about the population for each test.

You, presumably, do not know the ground truth for each and every one of your tests (or else you wouldn't be doing them in the first place). Therefore, I don't think you can form a strong expectation about the distribution of p-values you ought to see, other than: if for some portion of my tests, the null hypothesis in truth does not hold, then across all my tests I expect to reject the null more than 5% of the time (assuming p<0.05 is your criterion for rejecting the null).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.