# Interpretation when Information Criteria and F-Test Disagree

I used SPSS (v. 25) conducted a pair of nested multtlevel models (aka linear mixed models) where the outcome criterion is itself also nested within participants. I followed Singer and Willett's (2003) recommendations to include time as a level-1 (within-participant) predictor.

The "base" comparison model included no predictors except a fixed-effect (between-participant, level-2 factor) intercept and the random-effect time I mentioned just above.

The model I compared against that base model included those same two factors (i.e., intercept and time) and only added a fixed-effect, dummy-coded gender term.

The -2 log likelihood (-2 LL) for the base model was 28316. The -2 LL for the new extended model that added in gender was 27341. The deviance for this difference is 28316 - 27341 = 975. With 1 df, this difference is highly significant (critical χ2 = 5.02).

However, when reporting the inferential test for the gender term, SPSS indicates that the F-score for this term is 3.05, which is not significant (with dfs of 1 & 903.8 and α = .05).

In other words, adding the gender term made for a significantly better-fitting model. However, the term that was added that made for this better fit is itself not significant.

I am at a loss how to interpret this in practical terms. It seems that gender should be taken into consideration when understanding the outcome variable, but how do I reconcile explaining that gender itself is not significant?

Differences in deviance for a linear model can't just be compared to a $$\chi^2$$ reference distribution: the actual reference distribution is $$\sigma \chi^2$$, where $$\sigma$$ is the residual variance. If you knew $$\sigma$$, you could divide the deviance difference by it and compare to a $$\chi^2$$ distribution
The $$F$$ -test estimates $$\sigma$$ from the residuals, and divides the deviance difference by $$\sigma$$. To account for the information used up in estimating $$\sigma$$, you use an $$F$$ distribution -- but that doesn't matter much here since the .95 critical values for $$\chi^2_1$$ and $$F^1_903.8$$ are almost identical.
In your mixed model, things are slightly more complicated. The non-integer denominator df on the $$F$$ test are because the statistic doesn't have exactly an $$F$$ distribution. But the reason for the deviance difference and $$F$$-test to apparently disagree are the same