Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing)

It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery is, by definition, 0.05."

My question: I always thought false discovery is Type I error, which is equal to the chosen significance levels in most tests. p-value is the value calculated from the samples. Indeed Wikipedia states "The p-value should not be confused with the significance level α in the Neyman–Pearson approach or the Type I error rate [false positive rate]"

So why the statistician in the link claims Type I error is also the p-value?

share|improve this question
3  
Fisher's $p$ value and the Type I error rate $\alpha$ are incompatible according to the following paper: Hubbard, Bayarri (2012): Confusion over measures of evidence ($p$'s) versus errors ($\alpha$'s) in classical statistical testing. Also, have a look at this post here on the site. –  COOLSerdash Jul 4 at 6:31
1  
@COOL That's an awfully controversial paper to cite. Just take a look at the beginning of the discussion that follows on the last page. It seems to me the authors--willfully or unconsciously--misinterpret many of the statisticians they lambaste for being so ignorant and wrong. –  whuber Jul 4 at 16:06
    
@whuber This was my impression as well when I read the paper. Do you know a paper, book or post that offers a more benign treatment of this subject? –  COOLSerdash Jul 4 at 17:08
2  
@COOL There are so many I can't even keep track of them any more. –  whuber Jul 4 at 17:57
add comment

2 Answers 2

That sentence is rather confusing, especially out of context. It's notoriously hard to explain p values, and your source hasn't done so well. I can't edit the section at the source, but here's how I would:

The p-value is defined as the probability that we would get a test statistic as a big or bigger than the one observed if, in fact, the null is true of the population we sampled from, and we repeat the sampling procedure...When we conduct a single significance test, our error rate (i.e., the probability of having made a false discovery is just the p-value itself, given that the null is true (and that the "discovery" in question is the conclusion that the null hypothesis is untenable). For example, if we have a p-value of 0.05 and we conclude it is significant [irrelevant] the probability of a false discovery (i.e., an effect of at least equivalent size and reliability) in an exact replication of the given study is, by definition, 0.05. However, the more tests we do, the greater the probability of at least one false discovery (since these are independent events, to each of which the same probability applies).

It's a little clunkier, but don't blame me! Like I said, p values are hard to explain...and at least this version is correct. Anyway, I don't see it claiming that "Type I error is also the p-value," as you've read. That would be even further off.

share|improve this answer
    
I'm trying to digest it because I never knew conducting a single test, the false discovery rate is the p-value. –  Student T Jul 4 at 6:34
    
@ABCD: It's not. –  Nick Stauner Jul 4 at 6:36
    
You just stated in your answer it is! "When we conduct a single significance test...." –  Student T Jul 4 at 6:37
1  
Check again. There is a difference between "error rate" and the probability of replicating an effect in a given sample from a population in which the null is true. The p value represents the latter, upon which one might choose to base the conclusion that the null is untenable. Error rate does not involve a given sample's effect, among other differences. –  Nick Stauner Jul 4 at 6:40
add comment

Your false discovery rate not only depends on the p-value threshold, but also on the truth. In fact, if your null hypothesis is in reality wrong it is impossible for you to make a false discovery.

Maybe it's helpful to think of it like that: the p-value threshold is the probability of making false discoveries when there are no true discoveries to be make (or to put it differently, if the null hypothesis is true).

Basically,

Type 1 Error Rate = "Probability of rejecting the null if it's true" = p-value threshold

and

Type 1 Error Rate = False Discovery Rate IF the null hypothesis is true

is correct, but note the conditional on the true null. The false discovery rate does not have this conditional and thereby depends on the unknown truth of how many of your null hypotheses are actually correct or not.

It's also worthwhile to consider that when you control the false discovery rate using a procedure like Benjamini-Hochberg you are never able to estimate the actually false discovery rate, instead you control it by estimating an upper bound. To do more you would actually need to be able to detect that the null hypothesis is true using statistics, when you can only detect violations of a certain magnitude (depending on the power of your test).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.