I am confused by the dirac delta function in the formulation of monte carlo sampling. http://www.cs.ubc.ca/~arnaud/doucet_johansen_tutorialPF.pdf, for instance, defines in section 3.1 page 8 the marginal as $\pi(x_k)=\frac{1}{N}\sum_{i=1}^N\delta_{X^i_k}(x_k)$.

What does it mean here? That the density of $x_k$ is the average of the densities of the samples equal to $x_k$?

Thanks in advance.

  • $\begingroup$ Looks like $\delta$ is making the link between discrete and continuous. $X_k^i$ is in the discrete domain and the $\delta$ function defines an infinitely small interval around $x_k$ for the purpose of making $\pi$ a valid density. But still, I don't see what is the probability of $x_k$... $\endgroup$ – Mike Jan 23 '14 at 16:36
  • $\begingroup$ Perhaps my answer at stats.stackexchange.com/questions/73623/… addresses this question? $\endgroup$ – whuber Feb 25 '14 at 17:48

This is much simpler than you think. You need to understand that what underlies probability is integration. So, if you are interested in the marginal of the $k$th coordinate, what you really are looking for is knowing the value of $\int_A \pi(x_k) dx_k$ for all (measurable, whatever) $A$. How on earth would you estimate it? The solution is simple: you have a bunch of samples $X^{i} \in \mathbb{R}^d$. You estimate the above integral by counting how many times you have $X^{i}_k \in A$ and then divide by the number of samples you have - $N$. That is it.

In the notation you use, $\delta_{X_k^{i}}( \cdot)$ is a "function" s.t. if you integrate it over a set $A$, the result is 1if $X_k^i$ is in $A$ and zero otherwise: $\int_A \delta_{X_k^i}(x) dx = 1$ iff $X_k^i \in A$ and $0$ otherwise.

$$ \int_A \frac{1}{N}\sum_i \delta_{X_k^i}(x)dx = \frac{1}{N} \sum \int_A \delta_{X_k^i}(x )dx\\ =\frac{1}{N} \cdot \text{number of times $X_k^i$ is in $A$ } $$

just like our intuition would expect!!


Dirac delta function, is not exactly a function, but with a little abuse of notation you can consider $\delta_{x_0}(x)$ to be a function getting a non-zero value only at point $x=x_0$ and being a probability distribution, i.e. $\int_{-\infty}^{+\infty}=1$. Now you can extend this definition to multi-dimensional case with the same requirements. This is the case for Doucet et. al. tutorial. They are estimating the main distribution by a union of multi-dimensional dirac functions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.