1
$\begingroup$

I work in bioinformatics so I've seen my fair share of null multiple tests (figure 1) As well as a clear signal in p-value distributions (figure2) figure 1 figure 2

But I've also occasionally seen a type of p-value distribution with a high and low peak. enter image description here

I suspect this is a characterized event but I don't know what it is called so I cant look it up. Thinking about it, it seems to confound the False Discovery Rate's (FDR) ability to estimate the background uniform distribution causing no values to survive the multiple test correction though there clearly appears to be a significant signal.

So.

1) Is there a name for this phenomenon? What causes it? Is it enough to say that there are several inappropriate tests and can for that reason be ignored? Am I justified in trimming this long insignificant tail since these tests were clearly inappropriate?

2) Is there a multiple test correction robust to this phenomenon?

3) Is a large number of extremely insignificant p-values the same as a large number of moderately insignificant p-values: enter image description here

Is this a different phenomenon? Does it require a different response and/or multiple test correction in order to salvage the signal?

$\endgroup$
2
$\begingroup$

P-values are uniformly distributed under the null. In other words, if there is no true effect, you are equally likely to observe any p-value in the range $[0, 1]$.

On the other hand, when there is an effect, the p-values will tend towards zero (consider effect size).

Therefore, observing something like your third or fourth histogram suggests that something is done wrong. I would suggest to start with considering whether you are using the correct test, and whether its assumptions are well met.

The principle I would follow here is GIGO: garbage in garbage out. In other words, having "well formed" p-values will make the subsequent multiple testing corrections more reliable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.