# Confusion regarding p-values and false discovery rate

I get stuck with understanding of the following two statements (from Wikipedia on p-values):

1. The p-value is the probability of obtaining at least as extreme results given that the null hypothesis is true whereas the significance level $\alpha$ is the probability of rejecting the null hypothesis given that it is true.

2. If one defines a false positive rate as the fraction of all “statistically significant” tests in which the null hypothesis is actually true, several arguments suggest that this is at least about 30 percent for p-values that are close to 0.05.

It is more or less explained in the Regina Nuzzo's 2014 Nature editorial. I predefined a level of significance = 0.05, made a single test and got a p-value of 0.049. The second statement tells me that the chances that I will be able to replicate this result using another sample is not 95%, but much lower. (I think it should be dependent on prior probabilities, but the statement in wikipedia makes more general conclusion).

The questions are:

1. Is the second statement correct? Does it suggest that a priori probabilities of two hypothesis are equal to 0.5?

2. How to understand it intuitively?

• In your statement #2 it should read "false discovery rate" instead of "false positive rate". – amoeba Jan 14 '16 at 10:19
• see stats.stackexchange.com/questions/166323/…, section 1 is about p-values and section 2 about FDR – user83346 Jan 14 '16 at 10:32
• Why? Is it because the second statement suggests multiple testing? – German Demidov Jan 14 '16 at 10:33
• Any estimate of a false discovery rate (as opposed to calculating a rate conditional on the truth of some hypothesis) must involve assumptions about the prior probabilities of hypotheses. The assumptions may be plausible for some collections of tests (say those appearing in papers published in the journals of a particular field) but not for others. So unqualified statements like those in the Wikipedia article are unwise. [But now I look at the article in question I see that statement is followed by:-" In order to arrive at this number, one needs to postulate something about the prior ... – Scortchi - Reinstate Monica Jan 14 '16 at 12:30
• I see. I did not realize it's an exact quote. I have now formatted it as such. – amoeba Jan 14 '16 at 12:35

To answer your first question, for a set of tests providing $p$-values $\in [0.045,0.05]$ and power=0.8, the FDR (as defined in the second statement of your question) is 26% if there are as many tests with true effect as tests with no true effect (page 9 of the paper). Notice that the restriction to $p$-value $\in [0.045,0.05]$ is very important and the FDR decreases with letting the $p$-value having smaller values or/and when the proportion of true effect tests increases.