Let $Z \sim N(0,1)$ and let $Y=Z$. Suppose I wish to perform the following weird computation:
$f(z)=\int f(z|y)f(y)dy=E_Y[f(z|y)]$
and then use Monte Carlo to estimate $E_Y[f(z|y)]$. The problem is for a fixed $z$, the $y$ is draw is almost never equal to $z$, and so $f(z|y)$ is almost always zero, and so my Monte Carlo in practice will give me $E_Y[f(z|y)]=0$.
How should I address the fact that there is actually a probability $0$ that I will draw the much needed $y=z$ for which $f(z|y)$ is the infinite spike of strength $f(z)$, so that my Monte Carlo is actually supposed to work out?
In case you are wondering why I am doing such a weird "exercise", my actual context involves some knowledge of Dirichlet inference. I am trying to do something similar to slide 44 of this presentation, except with infinitely-many clusters. Basically, in a similar spirit to that slide, I would like
$f(\theta_{n+1} |X_1,...,X_n) \\ =\int f(\theta_{n+1} |\theta_1,...,\theta_nX_1,...,X_n) f(\theta_1,...,\theta_n|X_1,...,X_n) dX_1...dX_n \\ =E[f(\theta_{n+1} | \theta_1,...,\theta_n)]$
where $\theta_1,...,\theta_n \sim \theta_1,...,\theta_n | X_1,...,X_n$
I perfectly know how to use Gibbs Sampling to sample from $\theta_1,...,\theta_n | X_1,...,X_n$. However, I am worried about the fact that $f(\theta_{n+1} | \theta_1,...,\theta_n)$ contains dirac delta functions. Same concern as above: for a fixed $\theta_{n+1}$, I will almost never draw some $\theta_i$ equal to $\theta_{n+1}$. And if the dirac delta functions never come into play in my Monte Carlo, then $E[f(\theta_{n+1} | \theta_1,...,\theta_n)]$ with Monte Carlo will just be $h(\theta_{n+1})$ where $h$ was the distribution for the $DP$ - that is, no updating happened!!