How many p-value observations do you think are required before doing FDR correction

Question

I have two sets of correlation data and my variable of interest and I am wondering if/when o apply FDR to the obtained P-Values.

On one hand, I have correlations with gene expression data, so a great many deal of correlations (over 60k). lets call this dataset 1.

On the other hand, I have correlations with of my variable with smaller dataset of experimental outcomes from a different experiment. In here I have about 20-30 comparisons. Lets call this dataset 2

These experiments are independent so I am analysing them separately. For dataset 1 it makes sense to apply FDR. I would be afraid of type I error being present

However, it is less clear for dataset 2. I am afraid that i would be incurring in Type II errors if I were to apply FDR to a relatively small number of samples.

So my question is:

Is there a generally agreed limit of observations at which it makes sense to apply FDR to correct for type I? If so, is there any literature on it you can provide?

Answer 1

There are many contextual variables that should be considered before deciding to use any type of adjustment for multiplicity of comparisons. The direct answer to your question is therefore that there is no generally agreed limit and THERE SHOULD NOT BE ONE.

Here are a few things to consider:

1. The 'protections' by any adjustments against false positives come unavoidably at the cost of extra false negative results.

2. If your study is a preliminary or hypothesis generating one then you should not penalise any hypothesis or result for the presence of other unrelated hypotheses or results. Any 'false positives' will be found out in the follow up experiments with fresh data, and you do not want any extra false negatives that will be lost opportunities due to the lack of follow up.

3. There are many circumstances where a false negative result is as bad (or worse) than a false positive. (Read item 2 again.) Standard approaches to FDR and other 'corrections' for multiplicity privilege false positives over everything else.

4. If a discovery is interesting or important then it should be checked with a new experiment prior to it being accepted in anything other than a provisional manner. (You will not often see that written about in articles about statistical analysis.)

5. The accounting that underlies the 'corrections' for multiple comparisons relies on an all or none response to each dataset. That is not how a scientific analysis should respond to evidence.

Answer 2

Michael’s warnings are spot on. FDR is doing a lot of harm to research by getting researchers to ignore massive false non-discovery probabilities. FDR also completely fails at exposing the difficulty of the feature selection task. What really exposes the difficulty of the task, the reliability of the final result, and limitations due to non-huge sample sizes is getting confidence intervals of feature predictive importance. See here for a worked-out simulated example.

When you do more honest analyses that also do not use arbitrary thresholds for selecting features you’ll see that in many datasets all we know about “winning features” is that they are unlikely to be the worst losers.

Answer 3

TL;DR: you almost never want to "adjust" for a false discovery rate; that's not a real thing. "Adjusted" p-values are unintuitive, mostly reviled by statisticians as an inelegant hack, and bad summary statistics (as they're deeply unintuitive). What you really want is a simultaneous estimation approach or a multilevel model.

Full version: Suppose two researchers are trying to cure a disease. The first one does a trial testing one medication, gets a p-value of .01, writes up his results, and sends them off to a paper for publication. He repeats this process 10 times. The second researcher does the same experiments and gets the same results, but he does all 10 trials at the same time. Each of these trials individually has an individual p-value of .01, but he applies a Bonferroni correction. From the exact same data, we infer a p-value of 0.1 and cannot reject the null.

Here's my question to you: does this make even the tiniest bit of sense? Does the human body care whether you're publishing your results in one paper or two? No, of course not! This is insane!

But what about p-hacking and data dredging? If we test 100 different drugs and only one of them seems to work, chances are that one was probably a fluke. This is the intuition that misleads us into multiple comparisons.

It's a good intuition, but the wrong solution. Our intuition is saying that if the first 99 drugs failed, then before the experiment, we should think that the 100th probably won't work either--i.e. we assign a low prior probability that the 100th drug will work.

The correct way to resolve this is a random effects or hierarchical model. These models learn the correct prior from your data and partially pool estimates together. They're called random effects models because the drugs in each trial are no longer treated as completely independent. Instead, they're assumed to be random values that come from the same distribution; this lets us use the information from the first 99 drugs to predict how the 100th will go.

This completely solves the multiple comparisons problem. Not only that, it even improves your power at the same time! It's a free lunch.