I am confused with multiple comparisons adjustments. I have a $p$-values with lot of ones ( due to many scores in foreground are 0) from a fisher-exact test. I get some $p$-values which are significant without multiple testing correction. The $p$-value compose of 1000 $p$-values of

  Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 

0.0000013 0.2552000 0.6069000 0.5634000 0.8672000 .9900000

and 3000 $p$-values of 1. The $p$-values is present at https://dl.dropboxusercontent.com/u/2706915/pval.csv

If I remove all $p$-values=1 and perform multiple testing correction. I expected by adding these $p$-values=1; $q$-value will increase since distribution of $p$-value is shifting left. However, R-package pvalue functions are giving q-value=1 for all p-values. I cannot understand this behavior. The FDR assumes that p-value distribution is uniform that is not in my cases. What mistake I am making?.


1 Answer 1


A point by point response to your questions:

  1. You do not say what kind of test-statistic your $p$-values apply to. If you are talking about continuous distributions, such as for t or z statistics, then technically all of your $p$-values are strictly less than 1, although some of them may be very close to 1.

  2. You test a bunch of hypotheses, and some of them are significant (without multiple comparisons adjustments), and some of them are not. Great.

  3. Generally, one does not need to remove any $p$-values prior to conducting multiple comparisons adjustments for step-wise adjustment procedures (although the FDR gives the same results for a given level of $\alpha$). All but one adjusted $p$-value (i.e. $q$-values) will be always larger than the corresponding unadjusted $p$-value. Conversely, one can think of multiple comparisons adjustments as adjusting the rejection-probability (e.g. $\alpha$), rather than adjusting $p$-values, and here all but one of the adjusted rejection probabilities are less than the nominal type 1 error rate. One advantage to working the math out this way is one never has to adjust $p$-values so that they are larger than/truncated at the value 1.

  4. It sounds like, after adjustment for multiple comparisons using the FDR, you would not reject any hypotheses. This is a possibility (without seeing your vector of $p$-values it is not possible to show you the math).

  5. The FDR does not assume a uniform distribution of $p$-values.

  6. You are seemingly not making any mistake, other than being surprised by your results versus your expectations of your results.

Update: Have a look at this spreadsheet producing both adjusted alpha (i.e. the FDR), and alternatively adjusted $p$-values, for the 927 $p$-values in the spreadsheet you supplied.

Notice that: (1) column B contains the $p$-values <1 sorted largest to smallest; (2) column C contains the sorting order ($i$), (3) the adjusted $\frac{\alpha}{2} = \frac{0.05}{2}\times\frac{927+1-i}{927}$, (4) the adjusted $p$-values $=\frac{927}{927+1-i}p_{i}$, and finally, (5) you would reject the hypotheses corresponding to the two smallest $p$-values because (a) $3.78\times 10^{-5} < 5.39\times 10^{-5}$ (i.e. $p_{926} < \alpha_{926}^{*}$), or alternately (b) $0.0175 < 0.025$ (i.e. $q_{926} < \frac{\alpha}{2}$).

  • $\begingroup$ 1. p-value statistic is fisher exact test p-value. So if hit is 0 in foreground, p-value =1. 2. P-values is added to the question. $\endgroup$
    – avi
    Jul 6, 2014 at 21:14
  • $\begingroup$ You need to provide the actual $p$-values not summary statistics about the $p$-values. $\endgroup$
    – Alexis
    Jul 6, 2014 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.