Unintuitive results in model comparison I am running an experiment for some time now and currently I am in the process of analyzing my data. At first I was unsure about which model fits my needs, but after receiving some much appreciated help from the CV community, I decided that Mixed Effect Binomial Regression is the way to go.
In my experiment, each participant is assigned to a condition and then goes through 8 trials. In each trial, he/she is shown a stimulus and then I record whether they exhibit a certain behavior after that.
My model tries to predict Success (binary variable indicating whether they exhibited the target behavior, with values 0 and 1) using Condition (factor, with values experimental and control), Trial (numeric, with values 1-8), and Participant (string).
I have fitted two models so far, where the first one predicts Success using fixed effects for Condition, Trial, and a random effect for Participant. After fitting it, this is the sumary table I get in R:

As you can see nothing seems significant. However, it is also justifiable to include an interaction term between Condition and Trial into the model. After fitting this updated model, I get this summary table:

Now both the condition and the trial are significant. Furthermore, I did an anova to test whether there is any significant difference between these two models, and I got this:

So the situation is that there is no significant difference between these two models, however in one of them both of the predictors are significant! This looks very unintuitive to me, while it also makes unclear which model to select to base my analysis.
Do you agree that this seems a bit strange? Do you maybe have any ideas about the reasons behind it?
 A: *

*If you're running lots of tests, the probability that you occasionally find significant results even if nothing meaningful goes on is quite high.


*Andrew Gelman and Hal Stern show here...
https://www.tandfonline.com/doi/abs/10.1198/000313006X152649
...that "The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant". Even two models that have quite different assessments of the significance of certain parameters that occur in both of them may be so similar regarding their fit of the data that the data cannot decide between them, i.e., that the more complex one is not significantly better than the simpler one.


*Overall I don't think you have any indication that any of the significances you found is meaningful. (This doesn't mean that nothing is going on; you may test the first model against an uninformative intercept only model to see whether your variables together explain something even if the data are too weak to tell you which variable exactly does the trick. Also you may ask whether a potential trial effect may be nonlinear or non-monotonic; not sure what your trials are, but the numerical coding may be inappropriate in a generalised linear model.)
A: This is mostly an extended comment on the answer from @Christian Hennig (+1), which covers the critical points and which I think you should accept.
It's very tempting to think that a "significant difference" "supports" your hypothesis. That's not strictly how frequentist significance tests work. A small p-value argues against a null hypothesis, but (as pointed out in Point 2 of Christian Hennig's answer) it doesn't necessarily argue for any specific alternative hypothesis. A lack of "significance" doesn't mean the predictor is unimportant, just that your combination of data and model couldn't document its "significance" by an arbitrary (if widely used) criterion.
If (before seeing the data) you suspected that the association of success with Trial number might depend on Condition, then it would have made sense to start with a model containing the interaction. To evaluate if the fixed effects in that model add anything to a null model, compare that model against one with only the random intercepts--as Christian Hennig recommends in Point 3. If the null hypothesis of no fixed effects isn't tenable, then at least you have found something of potential interest. I suspect that you have, even in the model without the interaction term that showed neither predictor on its own to be "significant."
You are in a common situation where "the data are too weak to tell you which variable exactly does the trick," as he nicely put it. In that situation, attempts to improve the model with additional terms (splines, interaction terms, etc.) might not add enough to make up for the extra degrees of freedom that you use up, in terms of "significance."
Do, however, see whether the model makes sense in terms of your data. One way to interpret the interaction term coefficient of -0.230 for ConditionExperimental:Trial is in the context of the slope of +0.247 for Trial, which is the change per Trial in ConditionControl. That suggests an increasing probability of success with Trial in ConditionControl, but essentially no  increasing probability of success with Trial in ConditionExperimental (0.247 - 0.230 = 0.017). Is that how your data look? Does that make sense based on your knowledge of the subject matter? Comparing the data against the model is typically more useful than just staring at reported coefficient values.
Your results only seem counterintuitive because you are so highly focused on the arbitrary p = 0.05 cutoff for "significance." The interaction term might not be "significant" by that criterion, but that doesn't mean it's unimportant in the context of your study. This is one reason why many come to prefer Bayesian models that can provide credible intervals for parameters without the all-or-none fixation on arbitrary null-probability cutoffs.
