# An approach to analyse error in ABC posterior

I trying to understand approximate Bayesian computation (ABC) better so please evaluate my idea to analyse the error in the ABC posterior:

• Choose arbitrary constant values $\theta_0$ for model parameters $\theta$.
• Generate $n$ sets of synthetic data $X$.
• Use an ABC algorithm to compute an approximate posterior distribution $P(X|\theta)$ for each set of synthetic data $X_i$.
• There should be $n$ posterior distributions in total. For each posterior distribution $P_i(\theta|X)$ compute an empirical inverse cumulative distribution $F^{-1}(\theta)$
• Substitute the original constant model $\theta_0$ parameters into the inverse CDF $F^{-1}(\theta_0)$ and record the result for each posterior distribution $Y_i = F_i^{-1}(\theta_0)$.
• Draw a histogram of the $n$ samples of $Y$.

In the case that the posterior distribution is exact (no error) then $Y$ should be uniformly distributed on $[0,1]$. The tolerance parameter $\epsilon$ usually acts to widen the posterior distribution and hence in this case the distribution of $Y$ will be more concentrated in the centre of the $[0,1]$ interval. The deviation from a uniform distribution offers insights into the accuracy of the posterior distribution obtained through ABC.

• I'm interested to see where this question goes! But, the inverse CDF doesn't take quantiles as inputs -- it takes probabilities. – eric_kernfeld Jul 25 '17 at 20:45

There is no reason for the $F(\theta_i|X_i)$ to be uniformly distributed: it all depends on the initial distribution of the $\theta_i$'s. For instance, if they are generated from the prior, with cdf $\Pi(\cdot)$, the $\theta_i$'s transformed into $\Pi(\theta_i|X_i)$'s are uniform indeed. Else, not.

When using the ABC approximation to $\Pi(\cdot)$, there will be three types of approximations:

1. the one due to the non-zero tolerance $\epsilon$
2. the one due to the Monte Carlo error
3. the one due to the non-parametric error

and distinguishing between the three is delicate [at best].

A recent paper by Rodrigues, Prangle, and Sisson does exploit the $F^0\circ F_i^{-1}$ idea, albeit in a different fashion.

• I see now that $F(\theta_i|X_i)$ is influenced as well by the prior. Let me update my statement... Is it reasonable to say that the more non-informative the prior, the more sufficient the summary statistics and the smaller the tolerance parameter then the closer $F(\theta_i|X_i)$ will be to uniform? – Fundamental Engineer Jul 26 '17 at 12:36
• @JackPierce-Brown: stated in this in-quantitative manner, we can indeed say it should be closer to uniform. Although I do not think sufficiency is relevant: with insufficient statistics $\eta(X)$, the ABC distribution approximates $\pi(\theta|\eta(x))$. – Xi'an Jul 26 '17 at 15:47

Why is $Y_i$ uniformly distributed? I think I have worked up a simple example where that won't hold.

Suppose you are dealing with a simple model $X_i = \mu + Z_i$ where $\{Z_i\}$ consists of i.i.d. standard normals. You use a standard normal prior distribution on $X_i$ (call the density $\pi$). The posterior is then proportional to

$$L(\mu, X)\pi(\mu) \propto_\mu \exp(\frac{-\mu^2-\sum_i (X_i - \mu)^2}{2}) \propto_\mu \exp( \frac{-(N + 1)(\mu - \frac{\sum_i X_i }{ N + 1})^2}{2})$$

where $\propto_\mu$ means I can rescale by any function not involving $\mu$. This is the kernel of a Normal density with mean $\frac{\sum_i X_i }{ N + 1}$ and precision (inverse variance) $N + 1$.

Suppose I follow your scheme using the exact posterior instead of ABC, using 10 samples, and using a true parameter value of 1000 for $\mu$. I am likely to get a simulated $\sum_i X_i$ around 10,000. My posterior will center itself near 909, with almost no probability near the true value. With floating point arithmetic, the posterior CDF evaluated at the true value will always be 1.