# Bayesian MCMC when a likelihood function cannot be written

Consider Markov Chain Monte Carlo (MCMC) to sample from the posterior distribution of some unknown parameter $\theta$:

$$P(\theta|X) \propto P(X|\theta)P(\theta)$$

Where $X$ is the observed data, $P(X|\theta)$ is the likelihood function and $P(\theta)$ is the prior distribution.

My question concerns the case where an expression for $P(X|\theta)$ can not be written however for a given value of $\theta$ it is possible to numerically draw samples from $P(X|\theta)$.

How can we draw samples from $P(\theta|X)$? Links to literature would be appreciated.

My idiotic and computationally inefficient approach would be: for the current value of $\theta$ in the markov chain draw an arbitrarily large number of samples from $P(X|\theta)$ and fit an empirical likelihood function. However my intuition is that since the number of draws from $P(X|\theta)$ is arbitrary only one draw needs to be taken but it is not clear to me how this would work.

• Search terms that might be useful for you would be 'intractable likelihood' and 'doubly intractable posterior' and possibly 'approximate bayesian computation' – conjectures Mar 20 '17 at 16:07
• @conjectures or even just "how does MCMC work". – TrynnaDoStat Mar 20 '17 at 16:12
• @TrynnaDoStat: actually, sampling with an intractable likelihood function is a pretty advanced topic. Pretty sure googling "how does MCMC work" will not give the OP much insight on how to do that. I think you are referring to intractable posterior distributions. – Cliff AB Mar 20 '17 at 16:20
• The Wikipedia page: Approximate Bayesian computation seems to be an excellent starting point. – Fundamental Engineer Mar 20 '17 at 16:40
• see also Hartig et al., Statistical inference for stochastic simulation models - theory and application Ecol. Lett., 2011, 14, 816-827, onlinelibrary.wiley.com/doi/10.1111/j.1461-0248.2011.01640.x/… – Florian Hartig Apr 1 '17 at 8:26

The situation you describe where $p(x|\theta)$ cannot be computed but simulations from $p(\cdot|\theta)$ can be produced is call a generative model. It leads to likelihood-free resolutions like
2. synthetic likelihood, as in Wood (2010), where the unknown model $p(\cdot|\theta)$ is approximated by a Normal $\text{N}(\mu(\theta),\sigma(\theta)^2)$, where $\mu(\theta),\sigma(\theta)$ are estimated by simulation;