2
$\begingroup$

I'm using the abcpmc code:

Approximate Bayesian Computing (ABC) Population Monte Carlo (PMC) implementation based on Sequential Monte Carlo (SMC) with Particle Filtering techniques.

described in the Approximate Bayesian computation for forward modeling in cosmology, Akeret et al. (2015) article.

It seems to work rather well, but I'm not sure about the convergence/stopping criteria. Looking at the method's algorithm, it's easy to see that it can get stuck trying to find models below the given threshold ($\epsilon_t$):

enter image description here

where $N$ is the number of "particles" used. What I currently do is to stop the process if at any step it spends more than a fixed amount of time finding those $N$ particles below the current threshold value (say 30 sec per particle as the maximum allowed time for some 20 particles, ie: 10 min of max time at any step).

My questions are:

  1. is this a reasonable approach (not the particular values I use, but the general method)?
  2. As far as I understand, unlike the standard MCMC, with ABC one expects the acceptance rate to decrease as the process moves forward, ideally reaching a value as small as possible. Can this also be used as a stopping criteria (stop if acceptance rate is below some fixed value)?
  3. Are all the samples collected by an ABC sampler effective/independent samples (or are they correlated as with the usual MCMC)? Could this be used as another stopping criteria (stop at a given value of effective/independent samples)?
Improve this question
$\endgroup$
  • 1
    $\begingroup$ SMC-ABC selects the threshold $\epsilon_t$ based on the distance distribution, hence avoid the loops being stuck. $\endgroup$ – Xi'an Oct 15 '18 at 19:39
  • $\begingroup$ Thanks @Xi'an, I've been meaning to try pyABC which applies a ABC-SMC method. But in the meantime, are those convergence methods reasonable? $\endgroup$ – Gabriel Oct 15 '18 at 23:02
2
$\begingroup$

A first remark is that ABC only gets stuck when using predetermined tolerance levels. However there is rarely an instance where these levels have an absolute meaning, except for the limiting value $\epsilon_t=0$. They should thus be derived or calibrated from the prior predictive simulation itself.

  1. is this a reasonable approach (not the particular values I use, but the general method)?

A basic ABC method rejects a parameter value after one single failure to be near the observations, hence rejecting instead after N repeated failures or s seconds is equally fine.

  1. unlike the standard MCMC, with ABC one expects the acceptance rate to decrease as the process moves forward, ideally reaching a value as small as possible. Can this also be used as a stopping criteria (stop if acceptance rate is below some fixed value)

In their sequential ABC approach, Del Moral and Doucet (2011) stop decreasing $\epsilon_t$ when they cannot go deeper, i.e. when the acceptance rate gets too small. (We also advocated this approach in our population ABC.) This suggestion is similar.

  1. Are all the samples collected by an ABC sampler effective/independent samples (or are they correlated as with the usual MCMC)? Could this be used as another stopping criteria (stop at a given value of effective/independent samples)?

ABC samples are actually independent (and even iid) for the basic accept-reject schemes and some SMC implementations like ABC-PMC if not for ABC-MCMC versions. In the case of such importance sampling schemes, the importance weights can be turned into an effective sample size, as in usual importance sampling settings.

Improve this answer
$\endgroup$
  • $\begingroup$ Thank you for the detailed answer Xi'an. Two things in 3. are not entirely clear to me. "ABC samples are actually independent (...) for (...) some SMC implementation if not for ABC-MCMC versions". Does this mean that ABC-SMC is not considered an ABC-MCMC version but ABC-PMC is? And that samples are not independent in ABC-PMC? "I see no way to create an effective sample size", are not independent samples also effective samples? If not, what is the difference between independent and effective? I was under the impression they were synonyms. $\endgroup$ – Gabriel Oct 16 '18 at 12:41
  • $\begingroup$ ABC-PMC and ABC-SMC are not MCMC algorithms, but iterated importance samplers. You are right that the samples thus produced are not iid but weighted by importance weights, hence that effective sample size can be created. I thus deleted the last part of the answer! I do not understand the confusion between effective [meaning?] and independent. $\endgroup$ – Xi'an Oct 16 '18 at 14:55
  • $\begingroup$ So the samples produced by ABC-PMC are independent (I get this from your answer) but not iid (I get this from your comment)? Is this correct? If I understand correctly, the "effective sample size" (ESS) is a way to measure the number of "independent" samples in the chain. So if the ABC-PMC samples are independent, then the ESS is just the total number of (independent) samples collected? $\endgroup$ – Gabriel Oct 16 '18 at 15:26
  • 1
    $\begingroup$ I suggest you read first some introduction to importance sampling and effective sample size to catch the difference. For instance, we recently wrote a paper on the pros and cons of ESS in that context. $\endgroup$ – Xi'an Oct 16 '18 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.