How should an individual researcher think about the false discovery rate?

Question

I've been trying to wrap my head around how the False Discovery Rate (FDR) should inform the conclusions of the individual researcher. For example, if your study is underpowered, should you discount your results even if they're significant at $\alpha = .05$? Note: I'm talking about the FDR in the context of examining the results of multiple studies in aggregate, not as a method for multiple test corrections.

Making the (maybe generous) assumption that $\sim.5$ of hypotheses tested are actually true, the FDR is a function of both the type I and type II error rates as follows:

$$\text{FDR} = \frac{\alpha}{\alpha+1-\beta}.$$

It stands to reason that if a study is sufficiently underpowered, we should not trust the results, even if they are significant, as much as we would those of an adequately powered study. So, as some statisticians would say, there are circumstances under which, "in the long run", we might publish many significant results that are false if we follow the traditional guidelines. If a body of research is characterized by consistently underpowered studies (e.g., the candidate gene $\times$ environment interaction literature of the previous decade), even replicated significant findings can be suspect.

Applying the R packages extrafont, ggplot2, and xkcd, I think this might be usefully conceptualized as an issue of perspective:

Given this information, what should an individual researcher do next? If I have a guess of what the size of the effect I'm studying should be (and therefore an estimate of $1 - \beta$, given my sample size), should I adjust my $\alpha$ level until the FDR = .05? Should I publish results at the $\alpha = .05$ level even if my studies are underpowered and leave consideration of the FDR to consumers of the literature?

I know this is a topic that has been discussed frequently, both on this site and in the statistics literature, but I can't seem to find a consensus of opinion on this issue.

---

EDIT: In response to @amoeba's comment, the FDR can be derived from the standard type I/type II error rate contingency table (pardon its ugliness):

|                            |Finding is significant |Finding is insignificant |
|:---------------------------|:----------------------|:------------------------|
|Finding is false in reality |alpha                  |1 - alpha                |
|Finding is true in reality  |1 - beta               |beta                     |

So, if we are presented with a significant finding (column 1), the chance that it is false in reality is alpha over the sum of the column.

But yes, we can modify our definition of the FDR to reflect the (prior) probability that a given hypothesis is true, though study power $(1 - \beta)$ still plays a role:

$$\text{FDR} = \frac{\alpha \cdot (1- \text{prior})}{\alpha \cdot (1- \text{prior}) + (1-\beta) \cdot \text{prior}}$$

Answer 1

In order to aggregate the results of multiple studies you should rather think of making your results accessible for meta analyses. A meta analysis considers the data of the study, or at least its estimates, models study effects and comes to a systematical conclusion by forming some kind of large virtual study out of many small single studies. The individual $p$-values, ficticious priors and planned power are not important input for meta analyses.

Instead, it is important to have all studies accessible, disregarding power levels or significant results. In fact, the bad habit of publishing only significant and concealing non-significant results leads to publication bias and corrupts the overall record of scientific results.

So the individual researcher should conduct a study in a reproducible way, keep all the records and log all experimental procedures even if such details are not asked by the publishing journals. He should not worry too much about low power. Even a noninformative result (= null hypothesis not rejected) would add more estimators for further studies, as long as one can afford sufficient quality of the data themselves.

If you would try to aggregate findings only by $p$-values and some FDR-considerations, you are picking the wrong way because of course a study with larger sample sizes, smaller variances, better controlled confounders is more reliable than other studies. Yet they all produce $p$-values and the best FDR procedure for the $p$-values can never make up for quality disparities.

Answer 2

If I [individual researcher] have a guess of what the size of the effect I'm studying should be [...], should I adjust my $\alpha$ level until the FDR = .05? Should I publish results at the $\alpha=.05$ level even if my studies are underpowered and leave consideration of the FDR to consumers of the literature?

I would definitely not try to adjust the $\alpha$ level to reach a certain FDR, because it is very difficult: not only do you need to have a good estimate of power, but also a good estimate of the prevalence of nulls in some vaguely defined (!) population of studies that you imagine your own study to be part of. This is hardly possible.

On the other hand, even though I engaged in a long discussion with @DavidColquhoun about some specific claims in his paper, I do on some level agree with his practical recommendations in that $p<0.05$ does not strike me as a particularly strong evidence. Personally, I have learned to consider it as relatively weak, and am not convinced at all by many published results that hinge on a single $p\approx 0.05$. Truly convincing scientific results usually have either a tiny $p$-value $p\ll 0.05$, or are based on several experiments with supporting conclusions (such that a "combined" $p$-value would again be tiny).

So instead of adjusting your $\alpha$ in some specific automatic way, I would rather suggest to remain generally very skeptical about your own findings. Especially more so, if you know that your study is underpowered. Get more data. Think of a supporting analysis. Run another experiment. Etc.

Answer 3

This is actually a deep philosophical question. I'm a researcher myself and I've thought a while about this. But before an answer, let's review exactly what the false discovery rate is.

FDR versus P P is simply a measure of the probability of saying that there is a difference, when there is no difference at all and doesn't take the power into account. The FDR, on the other hand, takes the power into account. However, in order to calculate the FDR, we have to make an assumption: what is the probability that we receive a true positive result? That's something that we will never have access to, except under highly contrived circumstances. I actually spoke about this recently during a seminar I gave. You can find the slides here.

Here is a figure from David Colquhoun's paper on the topic:

The false discover rate is computed by dividing the number of false positives by the sum of the true positives and the false positives (in the example, 495/(80+495) x 100% = 86%!

A little bit more on P

Take a close look at the slides from my lecture. I discussed the fact that P values are drawn from a distribution. Which means that there will always be a chance that you will find a false positive. So statistical significance shouldn't be thought of as absolute truth. I argue that something that is statistically significant should be interpreted as, "Hey, there might be something interesting here, I'm not sure, someone go double check!" Hence, the fundamental notion of reproducibility in research!

So... what do we do? Well, an interesting point about the figure above and my analysis of P and FDRs is that the only way we can ever achieve a clear understanding is through 1) reproducibility and 2) publishing all results. That includes negative results (even though negative results are difficult to interpret). However, the conclusions that we draw from our results must be appropriate. Unfortunately, many readers and researchers do not fully understand the notions of P and FDR. I believe it is the responsibility of the readers to appropriately analyze results... which means that the burden is ultimately on the shoulders of educators. After all, a P value of 0.000000001 is meaningless if the "prevalence" (see figure above) is 0 (in that case, the false discovery rate would be 100%).

As a publishing researcher, just be careful to fully understand your results and make claims only as strong as you are willing. If it turns out that the FDR for your particular study is 86% (like the example above), then you should be very careful about your interpretations. On the other hand, if the FDR is small enough for your comfort.... still be careful about your interpretations.

I hope everything here was clear. It's a very important concept and I'm glad that you brought up the discussion. Let me know if you have any questions/concerns/etc.

Answer 4

To help understand the relationships, I created this graph of FDR as a function of prior probability for various powers (with alpha=0.05). Note this graph, and the equation of @Buckminster computes the FDR for all results with P less than alpha. The graph would look different if you only considered P values very close to the P value you happened to observe in one study.

Answer 5

To suggest publication is a decision. I think that it is worthwhile to study what are benefits and costs associated with this decision.

1) Academic environment universally pushes researchers to publish more, thought various rankings of publications will affect also this record. We can presume that more prestigious journals might have more robust quality checking (I hope so).

2) There might be social costs associated with the too large production of publications. These resources might be better used somewhere else, like in applied research without publications of results. There was recently a publication that many publications are not important as sources since sheer amount of new publications is so large... :)

http://arxiv.org/pdf/1503.01881v1.pdf

For the individual researcher number one forces to publish more and I think there should be institutionalized quality checks which are not dependent on individual peoples to keep quality at accepted level.

In any case your parameter values are not facts, these must be given values by the consideration of various costs and benefits associated with number of results published when results are truly and/or falsely significant.