Interpretation when Information Criteria and F-Test Disagree

Question

I used SPSS (v. 25) conducted a pair of nested multtlevel models (aka linear mixed models) where the outcome criterion is itself also nested within participants. I followed Singer and Willett's (2003) recommendations to include time as a level-1 (within-participant) predictor.

The "base" comparison model included no predictors except a fixed-effect (between-participant, level-2 factor) intercept and the random-effect time I mentioned just above.

The model I compared against that base model included those same two factors (i.e., intercept and time) and only added a fixed-effect, dummy-coded gender term.

The -2 log likelihood (-2 LL) for the base model was 28316. The -2 LL for the new extended model that added in gender was 27341. The deviance for this difference is 28316 - 27341 = 975. With 1 df, this difference is highly significant (critical χ² = 5.02).

However, when reporting the inferential test for the gender term, SPSS indicates that the F-score for this term is 3.05, which is not significant (with dfs of 1 & 903.8 and α = .05).

In other words, adding the gender term made for a significantly better-fitting model. However, the term that was added that made for this better fit is itself not significant.

I am at a loss how to interpret this in practical terms. It seems that gender should be taken into consideration when understanding the outcome variable, but how do I reconcile explaining that gender itself is not significant?

Answer 1

First, think of this just for a linear model, without any random term.

Differences in deviance for a linear model can't just be compared to a $\chi^2$ reference distribution: the actual reference distribution is $\sigma \chi^2$, where $\sigma$ is the residual variance. If you knew $\sigma$, you could divide the deviance difference by it and compare to a $\chi^2$ distribution

The $F$ -test estimates $\sigma$ from the residuals, and divides the deviance difference by $\sigma$. To account for the information used up in estimating $\sigma$, you use an $F$ distribution -- but that doesn't matter much here since the .95 critical values for $\chi^2_1$ and $F^1_903.8$ are almost identical.

In your mixed model, things are slightly more complicated. The non-integer denominator df on the $F$ test are because the statistic doesn't have exactly an $F$ distribution. But the reason for the deviance difference and $F$-test to apparently disagree are the same