Let $Z \sim N(0,1)$ and let $Y=Z$. Suppose I wish to perform the following weird computation:

$f(z)=\int f(z|y)f(y)dy=E_Y[f(z|y)]$

and then use Monte Carlo to estimate $E_Y[f(z|y)]$. The problem is for a fixed $z$, the $y$ is draw is almost never equal to $z$, and so $f(z|y)$ is almost always zero, and so my Monte Carlo in practice will give me $E_Y[f(z|y)]=0$.

How should I address the fact that there is actually a probability $0$ that I will draw the much needed $y=z$ for which $f(z|y)$ is the infinite spike of strength $f(z)$, so that my Monte Carlo is actually supposed to work out?

In case you are wondering why I am doing such a weird "exercise", my actual context involves some knowledge of Dirichlet inference. I am trying to do something similar to slide 44 of this presentation, except with infinitely-many clusters. Basically, in a similar spirit to that slide, I would like

$f(\theta_{n+1} |X_1,...,X_n) \\ =\int f(\theta_{n+1} |\theta_1,...,\theta_nX_1,...,X_n) f(\theta_1,...,\theta_n|X_1,...,X_n) dX_1...dX_n \\ =E[f(\theta_{n+1} | \theta_1,...,\theta_n)]$

where $\theta_1,...,\theta_n \sim \theta_1,...,\theta_n | X_1,...,X_n$

I perfectly know how to use Gibbs Sampling to sample from $\theta_1,...,\theta_n | X_1,...,X_n$. However, I am worried about the fact that $f(\theta_{n+1} | \theta_1,...,\theta_n)$ contains dirac delta functions. Same concern as above: for a fixed $\theta_{n+1}$, I will almost never draw some $\theta_i$ equal to $\theta_{n+1}$. And if the dirac delta functions never come into play in my Monte Carlo, then $E[f(\theta_{n+1} | \theta_1,...,\theta_n)]$ with Monte Carlo will just be $h(\theta_{n+1})$ where $h$ was the distribution for the $DP$ - that is, no updating happened!!


1 Answer 1


The problem in the first part of your question is wrongly formulated given that the conditional density does not exist.

According to slide 44, what you are interested in is the Posterior Predictive Distribution, which is a well-defined object in Bayesian inference.

Once you have a posterior sample, $\theta_1,\dots,\theta_N$, of the parameters of a model with density $f(\cdot\vert \theta)$ and a certain prior, this is a sample from $\theta\vert \text{Observations}$, then you can approximate the predictive distribution by:

$$\pi_{pred}(x_{n+1}\vert Observations)\approx\dfrac{1}{N}\sum_{j=1}^N f(x\vert\theta_j)$$

  • $\begingroup$ Hi, but what I really want is $\pi_{pred}(\theta_{n+1} | Observations)$ because $\theta$ is really my variable of interest and the $X$'s are just noisy observation of $\theta$'s. (same context as the slide 44) $\endgroup$ Aug 1, 2014 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.