In ABC sampling methods, Rejection, MCMC and SMC, when we sample potential parameter values from the prior/proposal, we then use those parameters on our model and simulate data values. This can be done without directly evaluating the likelihood (which is often difficult, expensive or impossible), and we then compare those simulated data values to the observed, and see if they are “close enough”, if so we accept the proposed parameter value. This is roughly the common idea behind these methods.

My question is this, besides simple models with clear distribution functions, how do we actually simulate data from complex models for a given parameter? For example if we are using the Lotka Volterra, SIR, HMM or a SDE model where directly computing the likelihood is impractical, how do we simulate data for a given parameter?

After reading about this topic, some suggest using numerical solvers in case of ODE models (eg Runge Kutta methods), but is this not the same as evaluating the model anyway?


1 Answer 1


This question is addressed in many ABC-related papers and in the Handbook of Approximate Bayesian Computation as well. The difference between a well-defined probability model $P_\theta$ and a closed-form likelihood is that the former only requires a generating process, which amounts to a function $$T: (0,1) \times \Theta \longmapsto \mathsf X$$ such that $T(U,\theta)$ is distributed from $P_\theta$ when $U\sim\mathcal U(0,1)$. This is for instance the principle behind the inverse-cdf generating method as $$X = F_\theta^{-1}(U) \sim P_\theta$$ when $U\sim\mathcal U(0,1)$. Deriving the density function of $P_\theta$ when this function $T(\cdot,\cdot)$ is known (either as a closed-form function or as an algorithm) is not always possible.

A common example in ABC papers is the g-and-k distribution, whose generating function and inverse cdf is $$X = F_{A,B,c,g,k}^{-1}(U) = A + B[1 + c \tanh(g U/2)]U(1 + U^2)k$$ while computing the density function requires a costly numerical inversion, as indicated in the gk R package. Nonetheless, the example remains artificial since an exact MCMC implementation is feasible, as pointed out by Pierre Jacob. And supported by his R code.

Another textbook level example is the case of a sample $\mathbf x$ from a standard distribution being solely observed through its median and median absolute deviation, $$ \mathfrak S(\mathbf x):=\{\text{med}(\mathbf x),\text{med}(|\mathbf x-\text{med}(\mathbf x)|)\} $$ While generating $\mathfrak S(\mathbf X)$ by first generating $\mathbf X$ is immediate and straightforward, deriving the density of $\mathfrak S(\mathbf X)$ is almost always impossible and the only known way to generate the posterior distribution of the model parameters given $\mathfrak S(\mathbf X)$ requires reconstructing the unobserved sample $\mathbf X$ which may prove too costly.

A much more realistic and convincing (and complex!) model is the one at the origin of the ABC algorithm, namely the most recent common ancestor model used in population genetics. In that model, it is straightforward to generate a current population of separated groups from the genetic characteristics of a common ancestor, following an binary evolution tree with random mutations along the branch, according to Kingman's coalescent model. In the simplest case with two current groups, the model only depends on a single parameter. Nonetheless, it is impossible to construct the likelihood of a sample from that model, no matter how small the sample is. This was the reason for the call to ABC by Tavaré et al. (1977) as they were unable to conduct inference on this model with MCMC or importance sampling methods.

In short, the difference between enjoying a generative model and accessing its density is a forward/backward or direct/inverse distinction.

  • 1
    $\begingroup$ I see thank you so much for clearing this up for me! This was the one detail in the ABC that was bothering me since the start. $\endgroup$
    – AlexS123
    Apr 14 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.