Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome.
What I'm doing is I'm fitting models predicting binary responses using two ordered factors A and B as predictors. I want to tell how much evidence I have that A and B interact (the effect of A gets stronger across the various levels of B).
I fit the models in brms
. I assume flat priors (because I have no assumptions, because I came over from frequentist model fitting, and because priors are scary) and a bernoulli error distribution and a logit link function.
First, I fit model A response ~ A + B
and model B response ~ A * B
.
Second, I compare the models using WAIC and ten-fold cross-validation (using loo_compare
).
Both these comparisons suggest that the predictive accuracy of model A is better than that of B -- or, in some cases, that it is not worse. (the elpd difference suggests better elpd for model A or the se of the difference eats the difference itself)
Given worse predictive accuracy for model B, I'd pick model A (and give up on the interaction).
Given no difference in predictive accuracy, I'm inclined to pick model A again, the model with no interaction, on account of parsimony.
Third, I ask: is model A a better fit because the data provide evidence for H0 (no ineraction) over H1 (interaction) or merely because they don't provide enough data to exclude H0?
I calculate Bayes factors (using bayes_factor(*model A*,*model B*)
, repeatedly, to see variance).
I expect one of two outcomes: BF will be 1< and high (meaning that model A is much more likely than model B). Or: BF will be ~1 (meaning that the two models are equally likely).
My result is that Bayes Factors prefer [edited] model B (with the interaction) to a massive degree.
I'd like to know how to interpret the conflicting evidence from predictive accuracy and Bayes Factors. It is entirely possible that picking a model is not the way to go.
If the result follows from a common mistake, I'd like to figure out what it is.