When I look at the definition of p value carefully:
$$ p = Pr(X<x|H_0) $$
where $H_0$ means the null hypothesis is true, thus condition negative. That having a test statistic $X<x$ means rejecting the hypotheiss, so I thought it matches the definition of false positive rate (FPR) based on the confusion table from Wiki, which is 1 - specificity.
Therefore, p-value + specificity = 1, and controlling p-value below a certain cutoff (i.e. $\alpha$) is equivalent to controlling specificity above 1 - $\alpha$. Is such reasoning correct? I feel uncertain because I haven't heard people discussing p-value together with specificity a lot.
Update (2018-05-23):
As pointed out by @Elvis, I made a conceptual mistake. It should've been
$$\alpha + \textsf{specificity} = 1$$
instead of p-value. Note that p-value is a random variable, while $\alpha$ and specificity are constants.
In other words, when we conduct a null-hypothesis test, we enforce the specificity of the test to be 1 - $\alpha$. This idea becomes obvious with a thought experiment.
To match the condition null-hypothesis being true, you take two samples from the same distribution, then you conduct a t-test. You do it for N times. If you set $\alpha=0.05$, then 5% of the times you would end up with a p-value < 0.05, thus rejecting the corresponding hypotheses. Since during each test, the samples are always drawn from the same distribution, so the null hypotheses are always true, and the rejected hypotheses result in type I errors. Therefore, the specificity is 0.95, the FPR is 0.05.
On a separate note brought up by @Tim, besides illustrating the sum of $\alpha$ and specificity being one, the above also demonstrates another problem that is p-value tells you nothing about FDR. In the above thought experiment, the false discovery rate (FDR) is 1, i.e. rejected hypotheses are all false positives/discoveries, because the prior probability is 0 (always null) regardless of the specificity. This issue has been discussed extensively. Besides @Tim's mention of a few posts, I recommend
- An investigation of the false discovery rate and the misinterpretation of p-values, David Colquhoun, R. Soc. open sci. 2014 1 140216; DOI: 10.1098/rsos.140216. http://rsos.royalsocietypublishing.org/content/1/3/140216
- Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
- Is Most Published Research Really False? Jeffrey T. Leek and Leah R. Jager, Annual Review of Statistics and Its Application 2017 4:1, 109-122, https://www.annualreviews.org/doi/10.1146/annurev-statistics-060116-054104
The first one is very readable. The second one exposes the seriousness of this problem. The third one tries to be more optimistic, but I thought the conclusion is still serious. Besides, I alos plotted the FDR as a function of sensitivity and specificty under different prior probabilities ($r$). The key insight is that When the prior probability is low, even when the specificity and sensitivity (less important) are high, the FDR could still be very high. For example, when the prior probability is 0.01, the specificity is 0.95 (corresponding to $\alpha = 0.05$ as commonly used) and the sensitivity is 0.8 (aka. power), the FDR is still as high as FDR. However, if $\alpha$ is set to 0.001, you would see a sharp drop in FDR as shown below (1st subplot). In general, $\alpha = 0.05$ is often too generous, which leads to much frustration in replicability and even abandoning p-values.
The plotted function is
$$\mathsf{FDR} = \frac{N (1 - r) (1 - \mathsf{specificity})}{N(1 - r)(1 - \mathsf{specificity}) + Nr\mathsf{Sensitivity}}$$
Details about plotting are provided in this Jupyter notebook.