WAIC and model selection

Question

I'm trying to compare 7 nested models at time. These are all Bayesian logistic regression estimated with stan. Some predictors will enter for sure in the model, while others have a polynomial structure. Thus I want to choose the polynomial degree. $$\ln\left(\frac{\pi}{1-\pi}\right)=\beta_0 +X\mathbf{\beta} + \beta_{n+1} z + \beta_{n+2}z^2+...$$ I'm using WAIC and LOO as criteria. In one case I have "clearly" different values:

     WAIC       LOO
 1   13067,6    13067,6
 2   13042,8    13042,9
 3   13016,2    13016,3
 4   12997,4    12997,5
 5   12999,9    13000,0
 6   12999,8    12999,9
 7   13000,7    13000,8

In the other one the difference among the values are just few decimals.

  WAIC      LOO
1 8408,2    8408,7
2 8408,8    8409,5
3 8407,7    8408,1
4 8408,1    8408,7
5 8407,3    8407,7
6 8407,8    8408,2
7 8407,9    8408,6

I was wondering if there is a rule to determine if the differences are large enough. I haven't found anything online. Clearly this are not the only tools I'm using. I also checked the convergence of the chains and other diagnostic tools and, in both cases, looking at them, I would exclude the models 5-6-7. But I don't know how to choose among the remaining ones.

Thanks in advance for your help.

Answer 1

See Section 5.2 in Aki Vehtari, Andrew Gelman and Jonah Gabry (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. In Statistics and Computing, 27(5):1413–1432. doi:10.1007/s11222-016-9696-4. arXiv preprint https://arxiv.org/abs/1507.04544 If you are using Stan with R, see loo package in CRAN https://cran.r-project.org/package=loo which computes the desired se.

Note also that if you have a large number of models, a direct comparison with LOO and WAIC is not recommended as the selection process will overfit. See more in Juho Piironen and Aki Vehtari (2017). Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3):711-735. doi:10.1007/s11222-016-9649-y. http://link.springer.com/article/10.1007/s11222-016-9649-y

For tutorial video and case studies for recommended way to do reliable and stable variable selection with Stan, see https://github.com/avehtari/modelselection_tutorial

To get faster answers for questions related to Stan and loo, I recommend Stan discussion forum http://discourse.mc-stan.org/