Bayes factors and predictive accuracy in model comparison in rstan / brms

Question

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome.

What I'm doing is I'm fitting models predicting binary responses using two ordered factors A and B as predictors. I want to tell how much evidence I have that A and B interact (the effect of A gets stronger across the various levels of B).

I fit the models in brms. I assume flat priors (because I have no assumptions, because I came over from frequentist model fitting, and because priors are scary) and a bernoulli error distribution and a logit link function.

First, I fit model A response ~ A + B and model B response ~ A * B.

Second, I compare the models using WAIC and ten-fold cross-validation (using loo_compare).

Both these comparisons suggest that the predictive accuracy of model A is better than that of B -- or, in some cases, that it is not worse. (the elpd difference suggests better elpd for model A or the se of the difference eats the difference itself)

Given worse predictive accuracy for model B, I'd pick model A (and give up on the interaction).

Given no difference in predictive accuracy, I'm inclined to pick model A again, the model with no interaction, on account of parsimony.

Third, I ask: is model A a better fit because the data provide evidence for H0 (no ineraction) over H1 (interaction) or merely because they don't provide enough data to exclude H0?

I calculate Bayes factors (using bayes_factor(*model A*,*model B*), repeatedly, to see variance).

I expect one of two outcomes: BF will be 1< and high (meaning that model A is much more likely than model B). Or: BF will be ~1 (meaning that the two models are equally likely).

My result is that Bayes Factors prefer [edited] model B (with the interaction) to a massive degree.

I'd like to know how to interpret the conflicting evidence from predictive accuracy and Bayes Factors. It is entirely possible that picking a model is not the way to go.

If the result follows from a common mistake, I'd like to figure out what it is.

Answer 1

Without more details, I can't speak about the specifics of this problem, but generally one should note that the ML that goes into BF is essentially the average likelihood over the prior, i.e. it is HIGHLY prior-dependent. WAIC and CV are calculated over the fitted model, and are thus approximately prior-independent, assuming that you have reasonably strong data.

Differences between BF and WAIC are therefore built-in when using arbitrary (in particular wide) priors. Logically, however, this is undesirable, as we would like to have consistent model selection. The basic solution to this is that you shouldn't use the BF on uncalibrated priors. Because you don't want to use your data twice, one option is to split the data and calibrate with one part, and calculate the BF on the hold-out (O'Hagan, Anthony. "Fractional Bayes factors for model comparison." Journal of the Royal Statistical Society: Series B (Methodological) 57, no. 1 (1995): 99-118.). If you do this, results should be roughly consistent with WAIC and CV.

For your specific case, I would further note that wide (or even inproper) priors usually favor the simpler model disproportionally when calculating the BF, so it is somewhat surprising that your BF favors model B stronger than WAIC with flat priors. This is a behavior that I would expect with narrow priors. Are you sure you have wide priors?

Answer 2

For the quizzical, this might be the way to go:

"The problem with your Bayes factor is that brms uses improper flat priors for regression coefficients by default. These priors should not be used with Bayes factors. Thus, you need to set reasonable and proper priors on these coefficients."

see: https://discourse.mc-stan.org/t/bayes-factors-in-brms/4109