Timeline for Confusion with false discovery rate and multiple testing (on Colquhoun 2014)
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 11, 2020 at 14:32 | history | edited | CommunityBot |
Commonmark migration
|
|
| Apr 22, 2015 at 7:16 | vote | accept | January | ||
| Apr 14, 2015 at 21:55 | history | edited | amoeba | CC BY-SA 3.0 |
rearranged everything
|
| Apr 14, 2015 at 14:45 | comment | added | amoeba | Okay, @Alexis, this is a good point. I guess I should revise my claim that "[controlling FDR across studies] would also be pretty impossible to do". But January's question, as I understand it, was explicitly about what happens when BH procedure is applied separately in each study (which mimics the actual scientific practice). This is what I was referring to in my answer (and in my previous comment). | |
| Apr 13, 2015 at 16:45 | comment | added | Alexis | Scalability as I understand it refers to combining sets of tests. Not separate applications: the FDR is not restricted to application under an arbitrary definition of "family". So I do not disagree with you last question because the point about scalability is the FDR is preserved when moving from 5 to 5000 tests (all together). | |
| Apr 13, 2015 at 16:38 | comment | added | amoeba | I just gave an extreme example, @Alexis. Here is a less extreme example: let's say you follow B-H procedure to control FDR on 5 tests and find some that are significant (under B-H) at alpha=0.05; then you do it again on another 5 tests. Then you repeat this 1000 times (on a total of 1000*5=5000 tests). Question: what is the FDR among all the tests you declared significant? Is it guaranteed to be below 5%? (As it would be if you run B-H on all 5000 tests together.) My point is that no, it is not. Do you disagree? | |
| Apr 13, 2015 at 16:32 | comment | added | Alexis | Well, what you describe is certainly not a multiple comparison procedure. However, performing FDR-based adjustment methods on, say 5 tests, and then adding 20 more to that set of 10 and performing the same method again preserves the rejection probabilities under FDR, but these rejection probabilities change under FWER. Dunn's Bonferroni adjustment provides a rather dramatic example. | |
| Apr 13, 2015 at 9:05 | comment | added | amoeba | @Alexis, I looked on wikipedia and it does say that FDR control is "scalable", but I don't know what exactly that is supposed to mean (I am not an expert). However, it is easy to see that if each paper has only one test performed, then Benjamini-Hochberg procedure does exactly nothing: it rejects if $p\le \alpha$ and accepts otherwise. Repeating this in many papers is equivalent to not using any FDR control and is certainly not equivalent to first collecting all the $p$-values across the papers, and then applying Benjamini-Hochberg procedure. | |
| Apr 12, 2015 at 23:14 | comment | added | Alexis | "But nobody ever adjusts for multiple comparisons across papers. It would also be pretty impossible to do." I thought one of the advantages of false discovery rate adjustments over familywise error rate adjustments was that while the latter require a definition of family, the former is scalable across an arbitrary number of comparisons? | |
| Apr 10, 2015 at 14:05 | comment | added | David Colquhoun | I'm not sure why @RyanSimmons says that I was "essentially just reiterating the well-known conceit that it is easy to find spurious effects in large sample sizes". It was nothing to do with large sample sizes! I'd really welcome an explanation of why he thinks the paper should have "a different (and less boldly stated) interpretation". | |
| Apr 10, 2015 at 13:12 | history | bounty ended | CommunityBot | ||
| Apr 9, 2015 at 10:43 | history | edited | amoeba | CC BY-SA 3.0 |
changed the wording
|
| Apr 8, 2015 at 17:26 | comment | added | Ryan Simmons | Granted, I only skimmed through the paper rather quickly, but it seems to me that he is essentially just reiterating the well-known conceit that it is easy to find spurious effects in large sample sizes (e.g. figure 1). Which isn't to say it isn't meaningful, but rather that I feel it should have a different (and less boldly stated) interpretation than the author provides. | |
| Apr 8, 2015 at 9:26 | history | edited | amoeba | CC BY-SA 3.0 |
fixed the quote
|
| Apr 2, 2015 at 21:54 | history | edited | amoeba | CC BY-SA 3.0 |
added some clarifications
|
| Apr 2, 2015 at 16:04 | history | answered | amoeba | CC BY-SA 3.0 |