# Leave one out cross validation in kernel density estimation

I am taking a look at : http://pages.cs.wisc.edu/~jerryzhu/cs731/kde.pdf

Where they define the following loss function for kernel density estimates

$$J(h) = \int \hat{f_n}^2(x)dx -2\int\hat{f_n}(x)f(x)dx$$ which comes from expanding the loss $$\int(\hat{f_n}(x)-f(x))^2dx$$ called the integrated square loss. This loss makes intiuitive sense to me because they are asking, how well did our kernel density match the true density.

However, I am unable to follow the next step. They claim we can re-write $J(h)$ as $$\hat{J(h)} = \int\hat{f_n}^2(x)-\frac{2}{n}\sum\hat{f_{-i}}(x_i)$$ meaning we approximate $J(h)$ with a leave-one-out approach (that is what the notation $f_{-i}$ means).

I really don't understand the intuition behind this. Can anyone help clarify?

Thanks!

• The identity (in expectation, anyways) in question was uncovered simultaneously in a pair of famous papers by Bowman and Rudemo in the early 1980s. Like many great ideas, it's obvious once you know it's true -- but it was not obvious to several generations of statisticians who came before then, including quite a few brilliant ones! So don't feel bad if you do not immediately see it.
– nth
Aug 21 '19 at 2:49

• Well, the point is to do cross-validation (i.e. the formula is not advertised as a training error?). As I understand it, the $\hat{f}_{-i}$ kernel-density excludes the basis function centered @ point $x_i$. Each term in the 2nd sum is the likelihood of a point under the kernel-density based on the other points, so when the average leave-one-out likelihood is high, the CV error is low. (For example, if a point is an outlier relative to its nearest-neighbor + bandwidth, it will have a low likelihood, so not contribute much to the sum). May 5 '17 at 12:53
• Well first of all, the sum is a full-sample average, just an average of $\hat{f}_{-i}$ rather than $\hat{f}_n$. In terms of bandwidth, the "missing terms" would just "want" $h\to{0}$, to put all the probability mass of their kernel $K_i(x)$ at their sample point $x_i$, no? May 5 '17 at 15:09