On the generality of using empirical FDR even for conservative p-value distributions Given the discussion here:
Why are p-values uniformly distributed under the null hypothesis?
And, particularly the point that @whuber brought up on the composite hypothesis (link here).
I'm wondering even if the distribution of p-values is not as good looking as it should, would it be fine to just empirically estimate the
   distribution of p-values (under null) or test statistics and
   calculate empirical FDR based on that?
So, this means that we can generalize this to all the cases where this happens?
Of course, it's always good to find the root of the problem. And, that this test is suffering from having Type I error less than selected $\alpha$. However, if one can get a good estimate of FDR by permutation on the expense of more computation, it's always a good idea, at least to me.
 A: Yes, that's a reasonable approach in principle. It's exactly what a permutation test does: estimate the distribution of any function under the null, then compare that to the value observed in the sample. It doesn't matter whether the function is called a "p value" or a "test statistic."
Permutation tests have a couple of limitations, though. You're limited to certain null hypotheses that can be simulated by permuting the data. Not all of them can be: see here for an example. Permutation test for model comparison? . Also, if you're using FDR corrections, maybe you're performing a lot of tests and you need a lot of precision. Permutation tests are rather granular by nature: if you have small sample sizes, the best p-value you can get is still kind of big. If you have large samples, then you can go smaller, but it's computationally expensive.
So, my question to you: if this question is prompted by a particular data analysis problem, then what is your null hypothesis, and how do you plan to simulate the distribution under it?
