Leave one out cross validation in kernel density estimation I am taking a look at :
http://pages.cs.wisc.edu/~jerryzhu/cs731/kde.pdf
Where they define the following loss function for kernel density estimates 
$$J(h) = \int \hat{f_n}^2(x)dx -2\int\hat{f_n}(x)f(x)dx$$
which comes from expanding the loss
$$\int(\hat{f_n}(x)-f(x))^2dx$$
called the integrated square loss. This loss makes intiuitive sense to me because they are asking, how well did our kernel density match the true density. 
However, I am unable to follow the next step. They claim we can re-write $J(h)$ as 
$$\hat{J(h)} =  \int\hat{f_n}^2(x)-\frac{2}{n}\sum\hat{f_{-i}}(x_i)$$
meaning we approximate $J(h)$ with a leave-one-out approach (that is what the notation $f_{-i}$ means). 
I really don't understand the intuition behind this. Can anyone help clarify?
Thanks!
 A: Comparing your formulas, we have
\begin{align}
J(h) & \approx \hat{J(h)} & \implies \\
\int\!\!\hat{f_n}(x)f(x)dx & \approx \tfrac{1}{n}\sum_i\hat{f}_{\!\!-i}(x_i) & \implies \\
\mathbb{E}\left[\hat{f_n}(x)\right] &\approx \overline{\hat{f}_{\!\!-i}(x_i)}
\end{align}
which says that the expected value of the full kernel estimate is approximately equal to the sample average of the "leave the evaluation point out" kernel estimate.
Does this help the intuition?
A: I think it is how leave-one-out cross validation works. Let's assume we have ten data points {x1,x2,...,x10}. Every time we train the bandwidth without only one point which will be used for validation purpose. I agree with you that we do the training and aim to minimize the error, but we may face an over-fitting issue and that's the reason why we apply cross-validation. Actually, it is full sample average; otherwise, the second term in J(h)^ should be divided by n-1 instead of n. In the ten data points example, we will train ten times and each time we have 9 training points and 1 validation point. Here n=10.
