Skip to main content
edited title
Link
amoeba
  • 107.2k
  • 36
  • 321
  • 346

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

Notice removed Draw attention by CommunityBot
Bounty Ended with amoeba's answer chosen by CommunityBot
tiny fix in formatting (along with edits in my answer)
Source Link
amoeba
  • 107.2k
  • 36
  • 321
  • 346

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as $30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the q$q$-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $30\%$, but below $5\%$, because I used Benjamini-Hochberg? That feels wrong, I would say that the $q$-value corresponds to the $p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a $q$-threshold of $0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in $30\%$ of the cases. However, I tried to explain it more formally and I failed.

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as $30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the q-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $30\%$, but below $5\%$, because I used Benjamini-Hochberg? That feels wrong, I would say that the $q$-value corresponds to the $p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a $q$-threshold of $0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in $30\%$ of the cases. However, I tried to explain it more formally and I failed.

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as $30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the $q$-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $30\%$, but below $5\%$, because I used Benjamini-Hochberg? That feels wrong, I would say that the $q$-value corresponds to the $p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a $q$-threshold of $0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in $30\%$ of the cases. However, I tried to explain it more formally and I failed.

added p-value tag (and made some minor formatting edits)
Source Link
amoeba
  • 107.2k
  • 36
  • 321
  • 346

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as 30%$30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the q-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not 30%$30\%$, but below 5%$5\%$, because I used BBenjamini-HHochberg? That feels wrong, I would say that the q$q$-value corresponds to the p$p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a q$q$-threshold of 0.05$0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in 30%$30\%$ of the cases. However, I tried to explain it more formally and I failed.

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as 30% even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the q-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not 30%, but below 5%, because I used B-H? That feels wrong, I would say that the q-value corresponds to the p-value in Colquhoun's paper and his reasoning applies here as well, so that by using a q-threshold of 0.05 I risk to "make fool of myself" (as Colquhoun puts it) in 30% of the cases. However, I tried to explain it more formally and I failed.

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as $30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the q-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $30\%$, but below $5\%$, because I used Benjamini-Hochberg? That feels wrong, I would say that the $q$-value corresponds to the $p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a $q$-threshold of $0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in $30\%$ of the cases. However, I tried to explain it more formally and I failed.

Tweeted twitter.com/#!/StackStats/status/583675559571501056
light editing
Source Link
amoeba
  • 107.2k
  • 36
  • 321
  • 346
Loading
Notice added Draw attention by January
Bounty Started worth 250 reputation by January
Source Link
January
  • 7.7k
  • 2
  • 37
  • 60
Loading