3
$\begingroup$

I perform several thousand tests on a genomic dataset (Mann Whitney U test). However, there is great variability in the sample size for each test (per test the number of measurements (independent) varies between 4 and 20 with outliers up to 75).

I understand that tests with more measurements (samples) have more power and are more likely to survive multiple testing (I use FDR at the moment). (i.e. the resulting list will be biased towards regions with larger sample size)

However, a reviewer pointed out to me that, apart from this bias, the p-values are "not comparable" and that standard adjustment procedures like FDR can not be used at all.

I understand the bias, but if I accept that (and describe it in the interpretation of the results as I do extensively), I do not see a problem.

$\endgroup$
4
$\begingroup$

I assume that by "FDR" you mean Benjamini-Hochberg. There are many FDR procedures, but in R the synonym for BH is "fdr", which is where often the confusion arises (also, B&H called their method "FDR controlling method").

As far as I know, the only important assumption is the independence of the hypotheses tested. There is nothing in the assumptions that states that the p-values for the hypotheses tested must be "comparable" (whatever the hell that is supposed to mean), or even that they should relate to the same type of statistical test. They are just hypotheses with associated $p$-values, that is all.

Here is the original Benjamini & Hochberg paper. The assumptions are, quote,

Consider testing $H_1, H_2, ..., H_m$ based on the corresponding $p$-values $P_1, P_2, \dots, P_m$. Let $P_{(1)} \le P_{(2)} \le \dots > \le P_{(m)}$ be ordered $p$-values, and denote by $H_{(i)}$ the null hypothesis corresponding to $P_{(i)}$.

I don't see a reference to sample size anywhere here, do you?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.