Timeline for After discovering alpha error inflation, how to estimate severity of the effect?
Current License: CC BY-SA 3.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 22, 2017 at 7:29 | comment | added | Ma Ba | @Glen_b your last comment is almost exactly the answer to my question "How can the total risk of both errors be quantified..".. I should be able to estimate the expected % of type I errors among all rejected nulls from the number of rejected nulls, the number of tests and the global p-value right? Similar thing could work for type II? could you add that to your answer? | |
| Sep 20, 2017 at 22:43 | comment | added | Glen_b | Note that if 100% of your nulls were true, you'd expect 15 rejections (all type I). If you had 100 rejections in the worst case you'd expect 11 of them would be type I errors. If you had 150 rejections in the worst case you'd expect 8 of them to be type I. At 200 rejections you'd expect about 5 type I errors under the worst case. Add in the "how plausible is a point null for this test" considerations and those (modest) numbers may be way too high; even without considering power you may not have much to worry about. | |
| Sep 20, 2017 at 22:12 | comment | added | Glen_b | ctd... which completely negate the need for looking at power, and there's little point worrying about any of this is you can't compare the costs (and approximate relative prior probabilities) of the null and various effect sizes at least in a roughly reasonable way (you might be able to consider some "best" and "worst" cases perhaps). But even if you can't get far with power analysis and those costs and probabilities, some of the other considerations will give some idea of whether there's all that much to worry about. | |
| Sep 20, 2017 at 22:11 | comment | added | Glen_b | I don't know that trying to conduct power analysis after the fact is necessarily the ideal approach and it's certainly not sufficient (although it would help to inform the considerations I was going on about). Where you can formulate appropriate ideas of effects it might be useful. But there's other considerations I mentioned - getting a rough upper bound on the number of type I errors (as per kodi's style of calculation); identifying implausible point nulls (e.g. "is it plausible males and females have identical population mean?" would probably be answered "well, no"), ... ctd | |
| Sep 20, 2017 at 20:18 | comment | added | Ma Ba | @Glen_b edited the question based on your comments. Yes, type II is important in this case! thats why I tried to clarify that I am not looking for p-value threshold adjustment methods. But generally the broad approach led to less useful p-values than they would have achieved in a more focused study right? I want to be able to give a quantitative indication of how much we can actually trust the reported results. Apart from that, do I understand correctly that I need to read up on power analysis and effect size? What other googable concepts are crucial in this scenario? | |
| Sep 20, 2017 at 19:50 | comment | added | Ma Ba | @Kodiologist so do I understand correctly that I need to use the total number of significant results if I want to quantify the issue? What would be the correct way to combine the number of tests, the pvalue threshold and the total number of rejected nulls to get a number that intuitively shows whether this is a problem or not? | |
| Sep 20, 2017 at 16:47 | comment | added | Glen_b | The OP calls the document a report. It doesn't sound like a journal article (but perhaps I misconstrue); imperatives may differ somewhat. Even if they didn't consider those things (and I agree they should have), the point in my answer is that it's not automatically the case that they did worse by (perhaps) blindly doing so than they would by blindly restricting the overall error rate to 5%. (I definitely understand that you're not suggesting that they should have done that either -- I don't see any substantive disagreement between us about what better practice would be.) | |
| Sep 20, 2017 at 16:41 | comment | added | David Ernst | I'm suggesting that if a paper does 300 comparisons, never mentions FWER corrections (not even to say why they shouldn't apply) and never mentions power considerations nor the trade-off between both issues, it is delusional to think they considered all those issues carefully but never wrote them down. Those authors careful enough to consider all these issues would know it is important to lay out their reasoning. If none of this can be found, it is safe to assume none of this was considered. There is still a separate question of how to tell them diplomatically that they screwed up. | |
| Sep 20, 2017 at 16:32 | comment | added | Glen_b | I'm not sure what you're suggesting there. Who do you think is deluding themselves in which way? | |
| Sep 20, 2017 at 16:30 | comment | added | David Ernst | Well aware of that and there is merit in not assuming the worst from people. However there is a trade-off, at a certain point you are just deluding yourself and harming everybody by ignoring obvious problems. (There might still be an issue with being able to speak truth to power, but that's separate.) | |
| Sep 20, 2017 at 16:26 | comment | added | Glen_b | No, the advice about considering type II errors was to the OP, not the people the OP worries have made an error in leaving the significance level at 5% for all tests. In considering type II errors (and their costs), one may be less inclined to insist that the overall type I error rate should be (say) 5% (giving a per test rate of say 0.00017 ... with corresponding effect on type II error) | |
| Sep 20, 2017 at 16:25 | comment | added | David Ernst | "In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power " Wouldn't that in most cases come down to, and rightly be perceived as, not a cautious response but a doubly scathing response: Not only did you inflate your type I errors, but on top of that you don't care about type II errors and have no clue about power analysis! | |
| Sep 20, 2017 at 16:23 | comment | added | Kodiologist | Fair enough. :) | |
| Sep 20, 2017 at 16:23 | comment | added | Glen_b | I don't think we'd be disagreeing over anything other than how much more "considerably more" is and how major "major" might be. | |
| Sep 20, 2017 at 16:21 | comment | added | Kodiologist | (+1) "If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections." — Suppose 24 of 100 null hypotheses are rejected. I agree that it's very unlikely that all those rejections are type-I errors, but isn't it still plausible that a substantial fraction of them are type-I errors? For example, 20 of the overall hypotheses might've been correctly rejected whereas the other 80 are true, and the remaining 4 rejections (5% of 80) are type-I errors. | |
| Sep 20, 2017 at 16:13 | history | edited | Kodiologist | CC BY-SA 3.0 |
edited body
|
| Sep 20, 2017 at 14:35 | history | answered | Glen_b | CC BY-SA 3.0 |