An approach to analyse error in ABC posterior I trying to understand approximate Bayesian computation (ABC) better so please evaluate my idea to analyse the error in the ABC posterior:


*

*Choose arbitrary constant values $\theta_0$ for model parameters $\theta$.

*Generate $n$ sets of synthetic data $X$.

*Use an ABC algorithm to compute an approximate posterior distribution $P(X|\theta)$ for each set of synthetic data $X_i$.

*There should be $n$ posterior distributions in total. For each posterior distribution $P_i(\theta|X)$ compute an empirical inverse cumulative distribution $F^{-1}(\theta)$

*Substitute the original constant model $\theta_0$ parameters into the inverse CDF $F^{-1}(\theta_0)$ and record the result for each posterior distribution $Y_i = F_i^{-1}(\theta_0)$.

*Draw a histogram of the $n$ samples of $Y$.


In the case that the posterior distribution is exact (no error) then $Y$ should be uniformly distributed on $[0,1]$. The tolerance parameter $\epsilon$ usually acts to widen the posterior distribution and hence in this case the distribution of $Y$ will be more concentrated in the centre of the $[0,1]$ interval. The deviation from a uniform distribution offers insights into the accuracy of the posterior distribution obtained through ABC.
Does this idea make sense? Is there already an established technique similar to this? If so then please link it. Thanks in advance for your help.
 A: Why is $Y_i$ uniformly distributed? I think I have worked up a simple example where that won't hold.
Suppose you are dealing with a simple model $X_i = \mu + Z_i$ where $\{Z_i\}$ consists of i.i.d. standard normals. You use a standard normal prior distribution on $X_i$ (call the density $\pi$). The posterior is then proportional to 
$$L(\mu, X)\pi(\mu) \propto_\mu \exp(\frac{-\mu^2-\sum_i (X_i - \mu)^2}{2}) \propto_\mu \exp( \frac{-(N + 1)(\mu - \frac{\sum_i X_i }{ N + 1})^2}{2})$$
where $\propto_\mu$ means I can rescale by any function not involving $\mu$. This is the kernel of a Normal density with mean $\frac{\sum_i X_i }{ N + 1}$ and precision (inverse variance) $N + 1$.
Suppose I follow your scheme using the exact posterior instead of ABC, using 10 samples, and using a true parameter value of 1000 for $\mu$. I am likely to get a simulated $\sum_i X_i$ around 10,000. My posterior will center itself near 909, with almost no probability near the true value. With floating point arithmetic, the posterior CDF evaluated at the true value will always be 1.
A: There is no reason for the $F(\theta_i|X_i)$ to be uniformly distributed: it all depends on the initial distribution of the $\theta_i$'s. For instance, if they are generated from the prior, with cdf $\Pi(\cdot)$, the $\theta_i$'s transformed into $\Pi(\theta_i|X_i)$'s are uniform indeed. Else, not.
When using the ABC approximation to $\Pi(\cdot)$, there will be three types of approximations:


*

*the one due to the non-zero tolerance $\epsilon$

*the one due to the Monte Carlo error

*the one due to the non-parametric error


and distinguishing between the three is delicate [at best].
A recent paper by Rodrigues, Prangle, and Sisson does exploit the $F^0\circ F_i^{-1}$ idea, albeit in a different fashion.
