The Fundamental Theorem of Simulation

Simulating $X \sim f$

is equivalent to simulating

$(X,U) \sim \mathscr{U}\{(x,u): 0<u<f(x)\}$

The proof is trivial.

One thing that is made clear by this theorem is that we can generate X in three ways:

  1. first generate $X \sim f$, and then $U|X=x$, but this is useless because we already have $X$ and don't need $X$
  2. first generate $U$, and then $X|U=u$, that is just the accept-reject algorithm
  3. generate $(X,U)$ jointly, which will eventually turn out be the smart idea because it allows us to generate data on a larger set were simulation is easier, and then to use the pair if the constraint is satisfied. http://academic.uprm.edu/wrolke/esma5015/fund.htm

I want to understand in last step,

we will sample $X$ using any method of MCMC and $U \sim U[0,1]$ and then take the sample of $X$ only which interested to find distribution for it.

Another question, Is this theorem related to latent variables?


1 Answer 1


Your question is unclear: when simulating $X\sim f(x)$, one can instead simulate$$(X,U)\sim\mathcal{U}(\{(x,u);\ 0<u<f(x)\})$$which has the joint density$$\mathbb{I}_{(0,f(x)}(u)\mathbb{I}_\mathcal{X}(x)$$and then use only the $X$ component in the simulation. For instance, here is a figure from our book depicting many realisations of such a Uniform when $f$ is the density of a Beta distribution [black dots]. The remaining dots are rejected simulations from a Uniform over a square that contains the subgraph of $f$.

Accept-reject simulation of a Uniform over the subgraph of a Beta density. Source: Robert and Casella (2004)

Once this new target is set, any simulation method aimed at it is valid. Like Accept-Reject in the above picture. What matters is to find an efficient simulation method. (The quote is then wrong in that $U$ is no longer $\mathcal{U}(0,1)$

As to your second question, the $U$ component in the above indeed behaves like a latent variable in the sense that $$f(x)=\int_0^\infty f(x,u)\text{d}u$$ represents the density of interest as the marginal density of a joint distribution of $(X,U)$, $U$ being "unobserved". However, latent variables are most often used in statistical settings where the joint $f(x,u)$ makes life easier, for instance allowing for an EM algorithm. In that sense $U$ is not a latent variable since this is not a statistical problem but a simulation problem. I would use "auxiliary" rather than "latent".

  • $\begingroup$ @ Xi'an The meaning of this joint distribution is unclear for me $(X,U) \sim \mathscr{U}\{(x,u): 0<u<f(x)\}$ .Can you explain it. I know that $$f(x)= \int du \,. $$ (the boundaries of the integral are between zero and f(x)) but this is a univariate distribution not Bivariate distribution. $\endgroup$
    – Manal
    Commented Jul 20, 2017 at 19:13
  • 1
    $\begingroup$ The joint distribution is a uniform distribution over the subgraph of $f$, which is a two-dimensional set when $X$ is univariate. Uniform distributions can be defined over any type of set with a finite measure (or volume). $\endgroup$
    – Xi'an
    Commented Jul 20, 2017 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.