There are a lot of questions on CV about this (e.g. here) but I am more interested in how much leeway I have to interpret these things.
I conducted an experiment where participants were asked to endorse a statement before and after an intervention. Some participants had experienced the intervention before. I wanted to test whether there would be any difference in the pre=post change in odds of endorsing the statement between those who had received the intervention before the study and those who hadn't. I conducted a repeated-measures logistic regression with training
(not trained
vs trained
) as the between-subjects factor, time
as the within-subjects factor (baseline
vs followup
), and participant id as the random factor.
Here are the proportions who endorsed the statement in each group at each time point
# training time endorsed count tot perc
# <fct> <fct> <fct> <int> <int> <dbl>
# 1 not trained baseline endorsed 40 65 61.5
# 2 not trained followup endorsed 62 65 95.4
# 3 trained baseline endorsed 26 29 89.7
# 4 trained followup endorsed 28 29 96.6
The overall interaction was non-significant, with the following odds ratios, 95% confidence intervals and p-value.
# or lowCI hiCI p
# 0.248 0.023 2.687 0.252
However the effect of time
in the not trained
group, who had not received the intervention before, was significant
# or lowCI hiCI p
# 12.949 3.565 47.040 0.000
Whereas the effect of time
in the trained
group, who had received the intervention before, was not significant
# or lowCI hiCI p
# 3.218 0.356 29.109 0.298
Now I understand how this can happen, how, to paraphrase Gelman, the difference between significant and non-significant can itself be non-significant. I assume in this case it is a numbers issue since the odds ratio of time even in the trained group, though non-significant, is still pretty respectable.
But what I do want to know is whether it is ok to go ahead and discuss this between group difference in the simple effects of time when there is no singificant omnibus interaction?
For example could you say say something like "despite the overall interaction being non-significant the intervention increased the odds of people endorsing the statement if they were experiencing it for the first time, however the odds of them endorsing the statement if they had received the intervention before did not increase with time?