# ABC model selection

It has been shown that ABC model choice using Bayes factors is not to be recommended due to the presence of an error coming from the use of summary statistics. The conclusion in this paper relies on the study of the behaviour of a popular method for approximating the Bayes factor (Algorithm 2).

It is well known that Bayes factors is not the only way for conducting model choice. There are other features, such as predictive performance of a model, that might be of interest (e.g. scoring rules).

My question is: is there a method, analogous to Algorithm 2, for approximating some scoring rule(s) or other quantities that can be used for conducting model choice in terms of the predictive performance in contexts with complicated likelihoods?

Nice question building on our work! Are you aware of the follow-up paper where we derive conditions on the summary statistic to achieve consistency in the Bayes factor? This may sound too theoretical but the consequence of the asymptotic results is quite straightforward:

Given a summary statistic $T$,

1. run an ABC algorithm based on $T$ for each model under evaluation ($i=1,..,I$) and estimate the parameters $\theta_i$ of those models by the ABC estimate $\hat\theta_i(T)$;
2. simulate the distribution of the statistic $T$ for each model and each estimated parameter, by a Monte Carlo experiment;
3. check whether the means $\mathbb{E}_{\hat\theta_i(T)}[T(X)]$ are all different by using step 2 with a sufficiently large number of iterations and, e.g., a t-test.

This procedure is not in the first version of the paper but should soon appear in the revised version

• Thanks for your answer. I was not aware of the second paper. It is an interesting result. A question that comes to my mind is the assumption of normality on the t-test (I know it is robust, but it might fail as well) together with the required significance level for a good approximation. Are you aware of other model comparison techniques with ABC? I remember a paper about DIC on arxiv. (I will assign the bounty in a couple of days to see if someone else shows interest in the question, I am not being mean :) )
– user10525
Apr 26, 2012 at 22:42
• Given that this is about my own paper/research, I do not really deserve a bounty, do I?! Apr 27, 2012 at 18:02
• It is a nice answer. I just have the feeling that you skipped twice the question about the existence of other techniques for measuring the predictive performance of a model using ABC. Even a 'yes, there exist' or a 'no, at least to my knowledge' would do the job.
– user10525
Apr 27, 2012 at 23:28
• About other methods, I would suggest taking a look at $\text{ABC}_\mu$ by Ratmann et al. (2009), which only considers the performances of each model within this model, comparing the observed errors with the distribution of the simulated errors. Not completely perfect, but very appealing nonetheless. Apr 29, 2012 at 7:39