# is empirical FDR denominated by measurements or by experiments?

I have a case-control experiment that produces one p-value of 0.002 for a given measurement (out of 100 measurements). If I shuffle the case-control labels and run the analysis 1000 times as a permutation test, 80 (or 8%) of my experiments will produce a p-value equal or lower to 0.002. In those 1000 pseudo-experiments, there were 100,000 measurements, but only 120 of them (or 0.12%) of them achieve this p-value.

Which is the real False Discovery Rate (8% or 0.12%)? What is the other rate called?

• I'm slightly confused, by shuffling them do you talk about permuting the entire vector? Because in that case, assuming you have a null hypothesis that is saying that some effect is non-existent, this null would be true (because your data is random), and as such whatever value you get can't really be the false-discovery rate, because the false-discovery rate has to be 100 %, since no correct discovery can be made. I might need a more detailed description
– Sam
Aug 30, 2017 at 17:29
• What you found out is that your test assigns a p-value of 0.002 or lower in 0.12% of the 100.000 random ($H_0$) cases, which is close to the expected 0.2% if the p-value belongs to a non-biased test. However, since you group those cases in 1000 batches of 100, the probability becomes much larger. And not coincidently $1-0.9988^{100} \sim 0.113$ is close to the 8% that you found. In summary, 8% is your estimated FWER, the probability that you have at least 1 error in 100 measuremen. And 0.12% is your estimated FDR, the expected fraction of measurements for which there is a type 1 error. Aug 31, 2017 at 13:59
• Delete that last sentence. 0.12% is the estimated fraction of the total measurements for which there is a type 1 error ($\neq$ FDR, which relates to the fraction of type 1 error among the group of rejections, and not the total). So you could say that 0.12% is the estimated p-value of your test (instead of the theoretic 0.2%). The discrepancy between the 0.12% and 0.2% might be due to violations of assumptions, for instance normality. Aug 31, 2017 at 14:08

The first value of 8% is just a p-value, relating to the combined null hypothesis that none of the 100 measurements are different between case and control.

You can reach this p-value by the following reasoning. Consider the null hypothesis that none of the measurement are different vs the alternative that at least one is different. Let's take the test statistic to be the minimum p-value obtained for any of the 100 measurements. You permute the labels, evaluate the test statistic for each permutation and the permutation p-value is 80/1000 = 0.08. (Actually, to be more precise, the permutation p-value is p = (80+1) / (1000+1), see Phipson & Smyth, 2010.)

The second value of 0.12% could be an estimated FDR under certain assumptions. To make this a valid estimator, you would need to be able to assume that the permutation distribution of p-values is the same for all of the 100 measurements that are not truly different between cases and controls. From the numbers you give, I suspect that this might not be a reasonable assumption.

It might be that some measurements are much less likely to yield small permutation p-values than others, for example because some measurements have more ties. If that is true, then the value of 0.12% is likely to underestimate the FDR.

Reference

Phipson, B., and Smyth, G. K. (2010). Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Stat. Appl. Genet. Molec. Biol. Volume 9, Issue 1, Article 39. https://doi.org/10.2202/1544-6115.1585

• Could you please expand on how the 0.12% could be considered an estimator for FDR? Sorry if it is intuitive but it's not clicking for me. Aug 31, 2017 at 12:24