8
$\begingroup$

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) it states

For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery is, by definition, 0.05.

My question: I always thought false discovery is Type I error, which is equal to the chosen significance levels in most tests. P-value is the value calculated from the sample. Indeed, Wikipedia states

The p-value should not be confused with the significance level $\alpha$ in the Neyman–Pearson approach or the Type I error rate [false positive rate]"

So why does the linked article claim that Type I error rate is given by the p-value?

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ Fisher's $p$ value and the Type I error rate $\alpha$ are incompatible according to the following paper: Hubbard, Bayarri (2012): Confusion over measures of evidence ($p$'s) versus errors ($\alpha$'s) in classical statistical testing. Also, have a look at this post here on the site. $\endgroup$
    – COOLSerdash
    Commented Jul 4, 2014 at 6:31
  • 2
    $\begingroup$ @COOL That's an awfully controversial paper to cite. Just take a look at the beginning of the discussion that follows on the last page. It seems to me the authors--willfully or unconsciously--misinterpret many of the statisticians they lambaste for being so ignorant and wrong. $\endgroup$
    – whuber ♦
    Commented Jul 4, 2014 at 16:06
  • $\begingroup$ @whuber This was my impression as well when I read the paper. Do you know a paper, book or post that offers a more benign treatment of this subject? $\endgroup$
    – COOLSerdash
    Commented Jul 4, 2014 at 17:08
  • 2
    $\begingroup$ @COOL There are so many I can't even keep track of them any more. $\endgroup$
    – whuber ♦
    Commented Jul 4, 2014 at 17:57
  • 1
    $\begingroup$ The linked article on surveyanalysis.org is garbage, and the quote is dead wrong. $\endgroup$
    – amoeba
    Commented Mar 10, 2017 at 11:27

2 Answers 2

Reset to default
9
$\begingroup$

Your false discovery rate not only depends on the p-value threshold, but also on the truth. In fact, if your null hypothesis is in reality wrong it is impossible for you to make a false discovery.

Maybe it's helpful to think of it like that: the p-value threshold is the probability of making false discoveries when there are no true discoveries to be make (or to put it differently, if the null hypothesis is true).

Basically,

Type 1 Error Rate = "Probability of rejecting the null if it's true" = p-value threshold

and

Type 1 Error Rate = False Discovery Rate IF the null hypothesis is true

is correct, but note the conditional on the true null. The false discovery rate does not have this conditional and thereby depends on the unknown truth of how many of your null hypotheses are actually correct or not.

It's also worthwhile to consider that when you control the false discovery rate using a procedure like Benjamini-Hochberg you are never able to estimate the actually false discovery rate, instead you control it by estimating an upper bound. To do more you would actually need to be able to detect that the null hypothesis is true using statistics, when you can only detect violations of a certain magnitude (depending on the power of your test).

Improve this answer
$\endgroup$
3
  • $\begingroup$ "A false discovery, Type I error, and false positive are all equivalent. Whereas the false positive rate and Type I error rate are equal, the false discovery rate is an entirely different quantity." A definition of FDR can be found here. $\endgroup$
    – Randel
    Commented Mar 21, 2016 at 21:11
  • 3
    $\begingroup$ Surely when the null hypothesis is true (or all null hypotheses are true) the FDR is by definition 100% (100% of all rejected null hypotheses are wrongly rejected). $\endgroup$
    – Björn
    Commented Mar 10, 2017 at 12:13
  • $\begingroup$ @Björn The FDR is a researcher choice, just like the FWER. If the FDR = 0.05 and all $m$ null hypotheses are true, then the expected number of false discoveries is $0.05m$. Nowhere in the seminal FDR literature will you see a method proposed for rejecting all true null hypotheses. I suspect you are confusing the ideas "all rejected true null hypotheses are false discoveries" and "the FDR rejects all true null hypotheses." $\endgroup$
    – Alexis
    Commented Mar 23, 2019 at 21:32
1
$\begingroup$

The difference between P values and false positive rate (or false discovery rate) is explained, clearly I hope, in http://rsos.royalsocietypublishing.org/content/1/3/140216

Although that paper uses the term False Discovery Rate, I now prefer False Positive Rate, becuse the former term is often used in the context of corrections for multiple comparisons. That's a different problem. The paper points out that for a single unbiased test, the false positive rate is a good deal higher than the P value under almost all circumstances.

There is also a qualitative description of the underlying logic at https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.