Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It has been shown that ABC model choice using Bayes factors is not to be recommended due to the presence of an error coming from the use of summary statistics. The conclusion in this paper relies on the study of the behaviour of a popular method for approximating the Bayes factor (Algorithm 2).

It is well known that Bayes factors is not the only way for conducting model choice. There are other features, such as predictive performance of a model, that might be of interest (e.g. scoring rules).

My question is: is there a method, analogous to Algorithm 2, for approximating some scoring rule(s) or other quantities that can be used for conducting model choice in terms of the predictive performance in contexts with complicated likelihoods?

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted
+50

Nice question building on our work! Are you aware of the follow-up paper where we derive conditions on the summary statistic to achieve consistency in the Bayes factor? This may sound too theoretical but the consequence of the asymptotic results is quite straightforward:

Given a summary statistic $T$,

  1. run an ABC algorithm based on $T$ for each model under evaluation ($i=1,..,I$) and estimate the parameters $\theta_i$ of those models by the ABC estimate $\hat\theta_i(T)$;
  2. simulate the distribution of the statistic $T$ for each model and each estimated parameter, by a Monte Carlo experiment;
  3. check whether the means $\mathbb{E}_{\hat\theta_i(T)}[T(X)]$ are all different by using step 2 with a sufficiently large number of iterations and, e.g., a t-test.

This procedure is not in the first version of the paper but should soon appear in the revised version

share|improve this answer
    
Thanks for your answer. I was not aware of the second paper. It is an interesting result. A question that comes to my mind is the assumption of normality on the t-test (I know it is robust, but it might fail as well) together with the required significance level for a good approximation. Are you aware of other model comparison techniques with ABC? I remember a paper about DIC on arxiv. (I will assign the bounty in a couple of days to see if someone else shows interest in the question, I am not being mean :) ) –  user10525 Apr 26 '12 at 22:42
1  
Given that this is about my own paper/research, I do not really deserve a bounty, do I?! –  Xi'an Apr 27 '12 at 18:02
    
It is a nice answer. I just have the feeling that you skipped twice the question about the existence of other techniques for measuring the predictive performance of a model using ABC. Even a 'yes, there exist' or a 'no, at least to my knowledge' would do the job. –  user10525 Apr 27 '12 at 23:28
1  
About other methods, I would suggest taking a look at $\text{ABC}_\mu$ by Ratmann et al. (2009), which only considers the performances of each model within this model, comparing the observed errors with the distribution of the simulated errors. Not completely perfect, but very appealing nonetheless. –  Xi'an Apr 29 '12 at 7:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.