How to choose the tolerance parameter for ABC?

Question

I have the following sorted data (sampling from parametric space [1,5]) with respect to their distances of parameter Theta. i.e., Let say N = 1000,

Theta : 1.1, 1.7, 1.9, 2.4, 2.8, . . . , 4.9

Distance : 0.2, 0.3, 0.5, 0.9, 1.1, . . . , 1.9

From literature I know that more than 3% (i.e., 0.03*N) data are not useful, but I don't know where to cutoff? Could you suggest any re-sampling methods? How can I treat this problem? Classification or Regression?

This data is basically output of rejection sampling (see http://en.wikipedia.org/wiki/Approximate_Bayesian_computation). Now I hope that would be clear.

Answer 1

One approach to choosing the cutoff value $\epsilon$ for ABC rejection sampling is the following (similar to Aniko's answer). Simulate several test data sets from known parameter values which are vaguely similar to your observed data (e.g. by performing ABC with a relatively large $\epsilon$). From the ABC output for a test data set, some criterion of performance compared to the true parameters can be calculated, such as mean squared error. Calculate this for all test data sets at many $\epsilon$ values, and choose $\epsilon$ to optimise the mean criterion (as this is a Monte Carlo estimate of its expectation). This requires many repetitions of the ABC algorithm, but can be done efficiently by using the same $N$ data simulations in every ABC algorithm (although this introduces some dependency between simulations).

In general, there is not a lot of published work on the choice of $\epsilon$. I think the approach above has been used somewhere and I will edit if I remember the references. An alternative is in "Choosing the Summary Statistics and the Acceptance Rate in Approximate Bayesian Computation" by Michael Blum. Other methods that I'm aware of apply only to SMC or MCMC methods.

Answer 2

Based on your edit, it appears that you are looking for guidance in selecting the tolerance parameter $\epsilon$ for ABC sampling. I don't know much about the topic, but $\epsilon$ should be small. A simple possibility is to select several different values and see whether the resulting posterior distributions look similar (based on new sets of samples). The largest value that still gives the same posterior can be used.