Possible to extend Gelman & Carlin's "Beyond Power Calculations" when significance not obtained?

Question

I have recently been spending time trying to understand the content and extent/limitations of this paper:

"Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" by Gelman and Carlin (2014)

http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf

In addition to the paper, I’ve found the thread linked, here on stat.stackexchange.com, to be helpful:

Understanding Gelman & Carlin "Beyond Power Calculations: ..." (2014)

I think my questions are, in concept, very similar, but different enough that the answers in that thread don’t answer the particular questions I have. Or perhaps my confusion doesn’t allow me to see that they already answer my questions.

It’s true that the Gelman and Carlin paper starts off by setting up the premise of the paper as “and you find a significant effect” as noted by Glen_b in the comments. However, are the methods as described in the paper limited to only that scenario?

Take an example of a comparison of two groups (A and B) evaluated by a t-test and consider these three scenarios:

(a) There was no alpha value chosen before the study, and the authors weren’t testing a p-value against an alpha, but just reporting a p-value (such as 0.06) and deciding that it was sufficiently small to conclude that they found an effect.

(b) There was an alpha value chosen (0.05), and the t-test didn’t reject the null because the p-value was 0.07. However, the authors conclude that there was likely an effect because the p-value was ‘close enough’ or ‘trending towards significance’ or some other similar language.

(c) There was an alpha value chosen (0.05), and the t-test didn’t reject the null because the p-value was 0.08. However, in addition to the t-test, the authors generated a Bayes Factor of 2.0 and claimed this showed that a difference between the two groups was twice as likely as having no difference between groups, and, therefore, conclude a difference in groups.

In none of those three scenarios was statistical significance achieved, but the conclusion from the authors was that there was a real effect demonstrated. For these scenarios, what is limiting the use of Type-S and Type-M calculations ala Gelman and Carlin?

In scenario (a), could we say that the authors implicitly chose an alpha of 0.06 or greater, since they were willing to say that the effect was real for p=0.06? In scenario (b) and (c), could we say the same thing as above, or perhaps use 0.05 (their official alpha) and say that the lower bounds for the Type-S and Type-M errors correspond to the p=0.05 calculations?

It seems like authors sometimes have an ‘official’ alpha value like in scenarios (b) and (c), and then when things don’t show a difference but are ‘close enough’ in their view, they conclude a difference anyway. I’d just like to know if we can utilize the reasoning in the Gelman and Carlin paper to estimate Type-S and Type-M errors in situations where statistical significance was not obtained but the researchers claim to have found an effect anyway.

I have a feeling that the answer by ely in the linked stackexchange thread above would do this very thing, but unfortunately the material is presently beyond my capabilities to understand. One day it’ll be within my reach, but today is not that day. Thank you.

Answer 1

Type M and Type S errors, as described in the paper, are for significant results, but there's nothing stopping the same methodology being used for non-significant results. You would just change the calculations to be conditional on non-significance, instead of significance.

The errors are the power rescaled in two different ways, so they have the same requirements as power calculations: a strict significance threshold, with a pre-determined $\alpha$, and a pre-determined value for the hypothetical "true" effect size to compare the null effect against. Like power calculations, these errors can be calculated before the study, as well as post-hoc.

In scenario (a), could we say that the authors implicitly chose an alpha of 0.06 or greater, since they were willing to say that the effect was real for p=0.06?

What we know about the authors' "real" choice of $\alpha$ is that it is at least as large as 0.06. Using this value to do the error calculations would give you a best-case value for the two types of error. However, as Figure 2 in the paper makes clear, if their value of $\alpha$ is larger than 0.06, this could result in substantially larger error values, especially type M. I would therefore treat these calculated error values with great scepticism, and that's without accounting for the possibility that the real $\alpha$ is highly dependent on the observed p-value.

In scenario (b) and (c), could we say the same thing as above, or perhaps use 0.05 (their official alpha) and say that the lower bounds for the Type-S and Type-M errors correspond to the p=0.05 calculations?

It seems like authors sometimes have an ‘official’ alpha value like in scenarios (b) and (c), and then when things don’t show a difference but are ‘close enough’ in their view, they conclude a difference anyway. I’d just like to know if we can utilize the reasoning in the Gelman and Carlin paper to estimate Type-S and Type-M errors in situations where statistical significance was not obtained but the researchers claim to have found an effect anyway.

There are two different conclusions you can analyse in this case:

1. The non-significant conclusion the authors should have made at their official value, $\alpha = 0.05$.
2. The significant conclusion they actually made, at $\alpha \geq 0.06$.

For conclusion 1, you can do the error calculations as usual. For conclusion 2., you're in the same case as in a), and could do error calculations using 0.06 as the lower bound, not 0.05.

However, because the two conclusions differ on whether the result is statistically significant, they will give very different results, and you can't take the error calculations for 1 and apply it to 2.

For example, because conclusion 1 is non-significant, the calculated type M error will be less than one, since non-significance removes large observed effects, in the same way that significance excludes small observed effects. (At least, for two-sided tests.) Applying the errors for conclusion 1 to conclusion 2 would suggest their significant observed effect is expected to be an under-estimate, while it is actually an over-estimate, potentially a very large one.

If the Bayes factor is calculated against a hypothetical "true" effect size, as in classical power calculations, I see no reason why scenario c) would differ from b), since the Bayes factor would be equivalent to the p-value as far as power calculations are concerned.