Monte Carlo evaluation of Integrated Square Error for Kernel Density Estimation

Question

Given the true density function $p(\mathbf{x})$ and the estimated density $q(\mathbf{x})$, I want to evaluate how close these two density are by Integrated Squared Error (ISE) criterion, $\int{(p(\mathbf{x})-q(\mathbf{x}))^2 dx}$.

In some $p$ and $q$, ISE can be derived analytically. However there are cases where it can not. I wonder if there is any Monte Carlo-style method for evaluate ISE in that case.

In other words, I want to evaluate the ISE integral with random points and function evaluations on the points.

I am especially concerned with the case where the support of $p(x)$ is infinite, for example Gaussian distribution. (Although for Gaussian $p$ and $q$, ISE can be evaluated exactly.)

Answer 1

One possibility for approximating the ISE is by expanding the binomial:

$$ISE = \int p(x)^2dx - 2\int p(x)q(x)dx + \int q(x)^2dx.$$

Assuming that you can simulate from $p$ and $q$, you can approximate this quantity as follows:

1. Simulate $N$ samples from $p$, $x_1,\dots,x_N$, and use the Monte Carlo approximation $\int p(x)^2dx \approx \frac{1}{N}\sum_{j=1}^N p(x_j)$.
2. Simulate $N$ samples from $q$, $y_1,\dots,y_N$, and use the Monte Carlo approximation $\int q(x)^2dx \approx \frac{1}{N}\sum_{j=1}^N q(x_j)$.
3. The cross product is a bit more tricky but you can compare the estimators:

$$2\int p(x)q(x)dx \approx \frac{2}{N}\sum_{j=1}^N p(y_j),$$

$$2\int p(x)q(x)dx \approx \frac{2}{N}\sum_{j=1}^N q(x_j).$$

If you cannot simulate from $p$ and $q$, then you may need use some deterministic numerical integration method such as quadrature. Another possibility is to use importance sampling integration, but you need to choose an appropriate importance density function $r$ that you can simulate from as follows:

$$\int p(x)^2dx = \int \frac{p(x)^2}{r(x)}r(x)dx \approx \frac{1}{N}\sum_{j=1}^N \frac{p(z_j)^2}{r(z_j)},$$ where $z_j\sim r$. Analogously for the other quantities. You can also use this trick on $(p-q)^2$.

Anyway, it is also a good practice to compare these estimators with those obtained with quadrature methods, and my main message is that life is not easy, and neither multivariate numerical integration.