3
$\begingroup$

This is from a typical introduction to kernel density estimation.

Suppose we want to estimate the probability density function $p(x)$ given a set of samples $x_1,x_2 \ldots x_N$. The simplest method that does this is from the histogram of the samples.

Divide the sample space into a number of bins and approximate the density at the center of each bin by the fraction of points in the training data that fall into the corresponding bin.

Suppose we have divided the sample space into $K$ bins of width $h$. The histogram of the given data is computed as :

$$ H(k) = \# \hspace{1mm} \mbox{of} \hspace{1mm} x^{(i)} \hspace{1mm} \mbox{in bin k } $$

This can be converted into a probability density as

$$ P(k) = \frac{H(k)}{N} $$

For a given sample $x$, let $p_k(x)$ be the probability that $x$ falls in bin $k$. From above, we have

$$ p_k(x) = P(k) $$

Why is it that the above is mentioned further normalized by bin width $h$ as follows ?

$$ p_k(x) = \frac{\# \hspace{1mm} \mbox{of} \hspace{1mm} x^{(k)} \hspace{1mm} \mbox{in same bin as x} }{N h} $$

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ This can be seen on dimensional grounds alone. Probability density is not probability, but a density with units probability per unit used. If the variable is length in mm, it is per mm; if the variable is age in years; it is per year. $\endgroup$
    – Nick Cox
    Commented Sep 24, 2014 at 12:08
  • $\begingroup$ Turns out there are some assumptions on the underlying $p(x)$ that bring in $h$. I found the following useful : research.cs.tamu.edu/prism/lectures/pr/pr_l7.pdf $\endgroup$
    – curryage
    Commented Sep 24, 2014 at 13:01
  • $\begingroup$ By definition a density is a probability divided by the bin width. See stats.stackexchange.com/questions/4220 for further discussion. This question therefore seems to have reversed the concepts of probability and density: $P(k)$ is the probability and $p_k(x)$ is the density. It is not valid to equate the two; as Nick Cox pointed out, they don't even have the same units of measurement. $\endgroup$
    – whuber ♦
    Commented Sep 24, 2014 at 16:49

1 Answer 1

Reset to default
6
$\begingroup$

The expression $$P(k) = \frac{H(k)}{N}$$ is the empirical relative frequency mass that falls into bin $k$, which we think of as empirical probability mass.

Let bin $k$ be the interval $[x_1, x_2]$, with center $x_k$. Then the estimated density curve $p(x)$ should be such that the area on this interval below the density curve is approximately $P(k)$. This can be approximated by a trapezoid, especially if bin length $h=x_2-x_1$ is "small". Applying the rule for the area of a trapezoid we have

$$ \frac{H(k)}{N}\approx (x_2-x_1)\frac {p(x_2) + p(x_1)}{2} \approx hp(x_k) $$

$$\Rightarrow \hat p(x_k)\approx\frac{H(k)}{Nh}$$

Sine we have started by remarking that $H(k)/N$ is empirical probability, this result also helps seeing why the density does not give probabilities.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Note that the trapezoid rule is not needed here and therefore invoking it may be an unnecessary distraction from the crux of the matter. Implicitly, KDE assumes the density is continuous. In this case the Mean Value Theorem of Calculus asserts the existence of $x_k\in [x_1,x_2]$ for which $$p(x_k)=\frac{1}{x_2-x_1}\int_{x_1}^{x_2}p(x)dx,$$ which is estimated by $\frac{1}{x_2-x_1}H(k)/N=H(k)/(Nh)$. These equalities are exact, not approximations (and $h$ need not even be "small"). $\endgroup$
    – whuber ♦
    Commented Sep 24, 2014 at 17:00
  • $\begingroup$ So $p_k(x)$ is an estimate of probability distribution. When $h$ is small, it close to probability $P(k)$. $\sum p_k(x) \neq 1$. $\sum P(k) = 1$. $\sum p_k(x) * h = 1$. Right? As the function hist() in R statisfies $\sum_i f^{(x[i])} (b[i+1]-b[i]) =1$, where $f^{(x[i])}$ is density in bins, $b[i+1]-b[i]$ is binwidth. $\endgroup$
    – Nick Dong
    Commented Mar 19, 2019 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.