I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including an effect provides clear evidence for an effect, but a comparison of that model to an intercept-only model via WAIC is inconclusive?
Some rudimentary simulations for context: I simulated datasets of varying size, all with a two-group design and a dichotomous outcome variable, where the probability of success was exactly .35 in one group and exactly .45 in another. The posterior predictions for the difference look like this:
(Note that N here refers to per-group N)
So, the model provides relatively unambiguous evidence for the difference at around N = 1,200. Using the same datasets, I then compared each of these models to corresponding intercept-only models, with these results:
(Where positive WAIC indicates an advantage for the effect model, and error bars = 1.96*SE of the WAIC difference.)
Now, even at N = 4,000, the (true) model with the effect is not clearly favored. Models were run in brms (family = binomial, link = logit) and compared via the loo_compare function. Priors were as follows:
Intercept ~ Normal(0,1)
b ~ Normal(0, 0.5)
I get that WAIC and other information criteria are not simply measuring effects in the same way posterior predictions are, but I'm still not sure how best to interpret findings like these. Is it the case that:
1) WAIC is overly conservative, and I'm better off interpreting model posterior predictions
2) Small effects are detectable in-model, but are unlikely to have an impact on predictive accuracy
3) There is some characteristic of my simulated data/analyses (i.e., binomial outcome, priors) affecting the WAIC comparisons
4) I'm misinterpreting WAIC model comparisons
5) Something else is going on
Out of these, 2) seems to make the most sense to me, but even when I run the same simulations with larger effects, the WAIC comparison doesn't favor the effects model until the sample size is 2-3x larger than samples in which the models themselves detect a difference.