One significant simple effect one non-significant simple effect but no significant interaction

Question

There are a lot of questions on CV about this (e.g. here) but I am more interested in how much leeway I have to interpret these things.

I conducted an experiment where participants were asked to endorse a statement before and after an intervention. Some participants had experienced the intervention before. I wanted to test whether there would be any difference in the pre=post change in odds of endorsing the statement between those who had received the intervention before the study and those who hadn't. I conducted a repeated-measures logistic regression with training (not trained vs trained) as the between-subjects factor, time as the within-subjects factor (baseline vs followup), and participant id as the random factor.

Here are the proportions who endorsed the statement in each group at each time point

#   training    time     endorsed count   tot  perc
#   <fct>       <fct>    <fct>    <int> <int> <dbl>
# 1 not trained baseline endorsed    40    65  61.5
# 2 not trained followup endorsed    62    65  95.4
# 3 trained     baseline endorsed    26    29  89.7
# 4 trained     followup endorsed    28    29  96.6

The overall interaction was non-significant, with the following odds ratios, 95% confidence intervals and p-value.

#    or lowCI  hiCI     p 
# 0.248 0.023 2.687 0.252 

However the effect of time in the not trained group, who had not received the intervention before, was significant

#     or  lowCI   hiCI      p 
# 12.949  3.565 47.040  0.000 

Whereas the effect of time in the trained group, who had received the intervention before, was not significant

 #    or  lowCI   hiCI      p 
 # 3.218  0.356 29.109  0.298 

Now I understand how this can happen, how, to paraphrase Gelman, the difference between significant and non-significant can itself be non-significant. I assume in this case it is a numbers issue since the odds ratio of time even in the trained group, though non-significant, is still pretty respectable.

But what I do want to know is whether it is ok to go ahead and discuss this between group difference in the simple effects of time when there is no singificant omnibus interaction?

For example could you say say something like "despite the overall interaction being non-significant the intervention increased the odds of people endorsing the statement if they were experiencing it for the first time, however the odds of them endorsing the statement if they had received the intervention before did not increase with time?

Answer 1

It's a matter of interpretation. You know one odds ratio is significantly higher than one (p < .001), and you know the other odds ratio is not significantly higher different from one (p = .298). You also know that these two odds ratios are not significantly different from one another (p = .252).

Put in another way: If we assume that the two odds ratios were the same, we'd expect to see this odds ratio difference (or a bigger difference) about 25% of the time.

You could totally say this, if you wanted:

Despite the overall interaction being non-significant, the intervention increased the odds of people endorsing the statement—if they were experiencing it for the first time; however the odds of them endorsing the statement if they had received the intervention before did not increase with time.

But you'd have to report that non-significant interaction. I personally would call this "weak evidence," but it depends on the conventions of your field or how much Type I and Type II error you are willing to tolerate.