I have recently been spending time trying to understand the content and extent/limitations of this paper:
"Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" by Gelman and Carlin (2014)
In addition to the paper, I’ve found the thread linked, here on stat.stackexchange.com, to be helpful:
I think my questions are, in concept, very similar, but different enough that the answers in that thread don’t answer the particular questions I have. Or perhaps my confusion doesn’t allow me to see that they already answer my questions.
It’s true that the Gelman and Carlin paper starts off by setting up the premise of the paper as “and you find a significant effect” as noted by Glen_b in the comments. However, are the methods as described in the paper limited to only that scenario?
Take an example of a comparison of two groups (A and B) evaluated by a t-test and consider these three scenarios:
(a) There was no alpha value chosen before the study, and the authors weren’t testing a p-value against an alpha, but just reporting a p-value (such as 0.06) and deciding that it was sufficiently small to conclude that they found an effect.
(b) There was an alpha value chosen (0.05), and the t-test didn’t reject the null because the p-value was 0.07. However, the authors conclude that there was likely an effect because the p-value was ‘close enough’ or ‘trending towards significance’ or some other similar language.
(c) There was an alpha value chosen (0.05), and the t-test didn’t reject the null because the p-value was 0.08. However, in addition to the t-test, the authors generated a Bayes Factor of 2.0 and claimed this showed that a difference between the two groups was twice as likely as having no difference between groups, and, therefore, conclude a difference in groups.
In none of those three scenarios was statistical significance achieved, but the conclusion from the authors was that there was a real effect demonstrated. For these scenarios, what is limiting the use of Type-S and Type-M calculations ala Gelman and Carlin?
In scenario (a), could we say that the authors implicitly chose an alpha of 0.06 or greater, since they were willing to say that the effect was real for p=0.06? In scenario (b) and (c), could we say the same thing as above, or perhaps use 0.05 (their official alpha) and say that the lower bounds for the Type-S and Type-M errors correspond to the p=0.05 calculations?
It seems like authors sometimes have an ‘official’ alpha value like in scenarios (b) and (c), and then when things don’t show a difference but are ‘close enough’ in their view, they conclude a difference anyway. I’d just like to know if we can utilize the reasoning in the Gelman and Carlin paper to estimate Type-S and Type-M errors in situations where statistical significance was not obtained but the researchers claim to have found an effect anyway.
I have a feeling that the answer by ely in the linked stackexchange thread above would do this very thing, but unfortunately the material is presently beyond my capabilities to understand. One day it’ll be within my reach, but today is not that day. Thank you.