2
$\begingroup$

This is from a typical introduction to kernel density estimation.

Suppose we want to estimate the probability density function $p(x)$ given a set of samples $x_1,x_2 \ldots x_N$. The simplest method that does this is from the histogram of the samples.

Divide the sample space into a number of bins and approximate the density at the center of each bin by the fraction of points in the training data that fall into the corresponding bin.

Suppose we have divided the sample space into $K$ bins of width $h$. The histogram of the given data is computed as :

$$ H(k) = \# \hspace{1mm} \mbox{of} \hspace{1mm} x^{(i)} \hspace{1mm} \mbox{in bin k } $$

This can be converted into a probability density as

$$ P(k) = \frac{H(k)}{N} $$

For a given sample $x$, let $p_k(x)$ be the probability that $x$ falls in bin $k$. From above, we have

$$ p_k(x) = P(k) $$

Why is it that the above is mentioned further normalized by bin width $h$ as follows ?

$$ p_k(x) = \frac{\# \hspace{1mm} \mbox{of} \hspace{1mm} x^{(k)} \hspace{1mm} \mbox{in same bin as x} }{N h} $$

$\endgroup$
  • 2
    $\begingroup$ This can be seen on dimensional grounds alone. Probability density is not probability, but a density with units probability per unit used. If the variable is length in mm, it is per mm; if the variable is age in years; it is per year. $\endgroup$ – Nick Cox Sep 24 '14 at 12:08
  • $\begingroup$ Turns out there are some assumptions on the underlying $p(x)$ that bring in $h$. I found the following useful : research.cs.tamu.edu/prism/lectures/pr/pr_l7.pdf $\endgroup$ – curryage Sep 24 '14 at 13:01
  • $\begingroup$ By definition a density is a probability divided by the bin width. See stats.stackexchange.com/questions/4220 for further discussion. This question therefore seems to have reversed the concepts of probability and density: $P(k)$ is the probability and $p_k(x)$ is the density. It is not valid to equate the two; as Nick Cox pointed out, they don't even have the same units of measurement. $\endgroup$ – whuber Sep 24 '14 at 16:49
5
$\begingroup$

The expression $$P(k) = \frac{H(k)}{N}$$ is the empirical relative frequency mass that falls into bin $k$, which we think of as empirical probability mass.

Let bin $k$ be the interval $[x_1, x_2]$, with center $x_k$. Then the estimated density curve $p(x)$ should be such that the area on this interval below the density curve is approximately $P(k)$. This can be approximated by a trapezoid, especially if bin length $h=x_2-x_1$ is "small". Applying the rule for the area of a trapezoid we have

$$ \frac{H(k)}{N}\approx (x_2-x_1)\frac {p(x_2) + p(x_1)}{2} \approx hp(x_k) $$

$$\Rightarrow \hat p(x_k)\approx\frac{H(k)}{Nh}$$

Sine we have started by remarking that $H(k)/N$ is empirical probability, this result also helps seeing why the density does not give probabilities.

$\endgroup$
  • 2
    $\begingroup$ Note that the trapezoid rule is not needed here and therefore invoking it may be an unnecessary distraction from the crux of the matter. Implicitly, KDE assumes the density is continuous. In this case the Mean Value Theorem of Calculus asserts the existence of $x_k\in [x_1,x_2]$ for which $$p(x_k)=\frac{1}{x_2-x_1}\int_{x_1}^{x_2}p(x)dx,$$ which is estimated by $\frac{1}{x_2-x_1}H(k)/N=H(k)/(Nh)$. These equalities are exact, not approximations (and $h$ need not even be "small"). $\endgroup$ – whuber Sep 24 '14 at 17:00
  • $\begingroup$ So $p_k(x)$ is an estimate of probability distribution. When $h$ is small, it close to probability $P(k)$. $\sum p_k(x) \neq 1$. $\sum P(k) = 1$. $\sum p_k(x) * h = 1$. Right? As the function hist() in R statisfies $\sum_i f^{(x[i])} (b[i+1]-b[i]) =1$, where $f^{(x[i])}$ is density in bins, $b[i+1]-b[i]$ is binwidth. $\endgroup$ – Nick Dong Mar 19 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.