# Empirical probability and Dirac distribution

According to Deep Learning p.65(Ian Goodfellow and Yoshua Bengio and Aaron Courville, available online):

(...) This can be accomplished by defining PDF using the Dirac delta function $\delta(x)$: $$p(x)=\delta(x-\mu)$$

(...) By defining $p(x)$ to be $\delta$ shifted by $\mu$ we obtain infinitely narrow and infinitely high peak of probability mass where $x=\mu$. A common use of Dirac delta distribution is as a component of an empirical distribution, $$\widehat{p}(x)=\frac{1}{m}\sum_{i=1}^{m}\delta\left(x-x^{(i)}\right)$$(where $x^{(i)}$ are our data empirical datapoints).

My questions are:

1) Can we think of $\widehat{p}(x)$ as a kernel density estimator with the kernel $K()$ defined as the Dirac function at zero?

2) How can we calculate the mean of Dirac function?

3) What is the intuition behind the above empirical probability distribution $\widehat{p}(x)$?

4) If $x=x^{(i)}$ then $\widehat{p}\left(x=x^{(i)}\right)=+\infty$?

• (1) and (3) are answered at stats.stackexchange.com/questions/73623. (2) is trivial: when all the probability is assigned to the value $\mu$, the mean obviously is $\mu$. (4) expresses the intuition given in the question, but is not mathematically correct: $\delta$ is actually a probability measure. It does not really have a PDF. – whuber Nov 13 '16 at 20:41
• So generally we can say that dirac function and dirac PDF is a ''concept'' used to define derivative of empirical cummulative density function, without interpretation itself? – mokebe Nov 14 '16 at 10:10
• The Dirac delta does have an interpretation, either as a measure or a generalized function. But attempts to interpret it as a probability density are risky, because they can lead to mathematically invalid conclusions. – whuber Nov 14 '16 at 14:42
• Minor correction: A Dirac measure $\delta_{x_0}$ has a density against any measure that puts positive mass on $x_0$. Therefore, the density of $\hat{p}$ against the counting measure on the set $\{x^ {(1)},\ldots,x^ {(n)}\}$ exists and is $1/n$ at each of the $x^ {(i)}$' s and zero elsewhere. – Xi'an Nov 21 '16 at 8:57
• @Xi'an When I wrote the first comment I figured somebody might come along with that correction--but I was hoping they wouldn't. :-) – whuber Nov 21 '16 at 20:18

As noted by whuber your question was answered to the great extent in his answer to another question on similar topic.

1) Can we think of $\widehat{p}(x)$ as a kernel density estimator with the kernel $K()$ defined as the Dirac function at zero?

If you have some kernel $K$ parametrized with bandwidth $h$, then as $h \to 0$ your kernel becomes Dirac delta. This is basically one of the intuitions behind Dirac delta as also described on Wikipedia.

2) How can we calculate the mean of Dirac function?

It accumulates all probability mass at $\mu$ and is zero elsewhere, so from the definition of expected value is $\mu$.

3) What is the intuition behind the above empirical probability distribution $\widehat{p}(x)$?

It's a discrete distribution, you assume that your variable can take only the values as observed in your sample (i.e. $x$'s), with probabilities equal to empirical probabilities. If your data is not discrete, then only empirical cumulative distribution function makes sense as an approximation of the true distribution. This is why histograms and kernel density estimators were developed, so that we can approximate the concinnous density function using empirical data.

4) If $x=x^{(i)}$ then $\widehat{p}\left(x=x^{(i)}\right)=+\infty$?

Dirac delta is not a function and is not infinite as $x^{(i)}$. This is just a heuristic about it, that is meant to provide an intuitive understanding of what it is, but if it was infinite, it would be useless. If you want to learn more, check Role of Dirac function in particle filters thread, where it is discussed in greater detail.

If you want intuitive understanding of it, then think of a $m$ distributions, where each of them integrates to unity and has the same probability density mass at $x^{(i)}$'s. You create a mixture of those distributions, each appearing with equal mixing proportion $1/m$. So whatever was the probability density mass at $x^{(i)}$'s, first you divide it by $m$, then multiply by $m$ (there was $m$ $\delta$'s), so in total you need to end up with same total mass equal to unity, and mass for each of the $x^{(i)}$ points would be $m$ times smaller then initially. Probability densities for $x^{(i)}$'s are huge since the area under each of the points is infinitesimally small (see Can a probability distribution value exceeding 1 be OK? to learn more about probability densities). It gets abstract and dirty, but the take-away message is that it is just a trick to use calculus with discrete values. If you stay at discrete-only reality, then basically each of the $x^{(i)}$ points has mass equal to $1/m$ so that in total their mass sums to unity by the axioms of probability.

• +1. Concerning (4), it seems one really could argue that $\hat p(x^{(i)})=\infty$ in a well-defined sense. One way is to take the limit of approximate probability densities $$\frac{1}{\mu(\mathcal{R})}\int_\mathcal{R} \hat p(x)dx$$ for a net of measurable neighborhoods of $x^{(i)}$ converging to $x$, each with Lebesgue measure $\mu(\mathcal{R})$. By definition of the Dirac delta, all of those integrals equal $1$, whence (since $\mu(\mathcal{R})\to 0$) those probability densities eventually exceed any integer $N$: that is, they converge to $+\infty$ in the extended Reals. – whuber Jan 30 '17 at 16:24