4
$\begingroup$

Given the discussion here: Why are p-values uniformly distributed under the null hypothesis?

And, particularly the point that @whuber brought up on the composite hypothesis (link here).

I'm wondering even if the distribution of p-values is not as good looking as it should, would it be fine to just empirically estimate the distribution of p-values (under null) or test statistics and calculate empirical FDR based on that?

So, this means that we can generalize this to all the cases where this happens?

Of course, it's always good to find the root of the problem. And, that this test is suffering from having Type I error less than selected $\alpha$. However, if one can get a good estimate of FDR by permutation on the expense of more computation, it's always a good idea, at least to me.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
+50
$\begingroup$

Yes, that's a reasonable approach in principle. It's exactly what a permutation test does: estimate the distribution of any function under the null, then compare that to the value observed in the sample. It doesn't matter whether the function is called a "p value" or a "test statistic."

Permutation tests have a couple of limitations, though. You're limited to certain null hypotheses that can be simulated by permuting the data. Not all of them can be: see here for an example. Permutation test for model comparison? . Also, if you're using FDR corrections, maybe you're performing a lot of tests and you need a lot of precision. Permutation tests are rather granular by nature: if you have small sample sizes, the best p-value you can get is still kind of big. If you have large samples, then you can go smaller, but it's computationally expensive.

So, my question to you: if this question is prompted by a particular data analysis problem, then what is your null hypothesis, and how do you plan to simulate the distribution under it?

Improve this answer
$\endgroup$
2
  • $\begingroup$ What convinced you that hypotheses of exactly zero effect are of interest? $\endgroup$
    – Frank Harrell
    Oct 21, 2017 at 15:11
  • 2
    $\begingroup$ I discussed permutation tests because the question mentions them. It's good that you mention that, though -- it brings up another important class of null hypotheses, different from the example I gave in my answer, that permutations tests also cannot address. $\endgroup$
    – eric_kernfeld
    Oct 22, 2017 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.