0
$\begingroup$

Given the true density function $p(\mathbf{x})$ and the estimated density $q(\mathbf{x})$, I want to evaluate how close these two density are by Integrated Squared Error (ISE) criterion, $\int{(p(\mathbf{x})-q(\mathbf{x}))^2 dx}$.

In some $p$ and $q$, ISE can be derived analytically. However there are cases where it can not. I wonder if there is any Monte Carlo-style method for evaluate ISE in that case.

In other words, I want to evaluate the ISE integral with random points and function evaluations on the points.

I am especially concerned with the case where the support of $p(x)$ is infinite, for example Gaussian distribution. (Although for Gaussian $p$ and $q$, ISE can be evaluated exactly.)

$\endgroup$
3
$\begingroup$

One possibility for approximating the ISE is by expanding the binomial:

$$ISE = \int p(x)^2dx - 2\int p(x)q(x)dx + \int q(x)^2dx.$$

Assuming that you can simulate from $p$ and $q$, you can approximate this quantity as follows:

  1. Simulate $N$ samples from $p$, $x_1,\dots,x_N$, and use the Monte Carlo approximation $\int p(x)^2dx \approx \frac{1}{N}\sum_{j=1}^N p(x_j)$.
  2. Simulate $N$ samples from $q$, $y_1,\dots,y_N$, and use the Monte Carlo approximation $\int q(x)^2dx \approx \frac{1}{N}\sum_{j=1}^N q(x_j)$.
  3. The cross product is a bit more tricky but you can compare the estimators:

$$2\int p(x)q(x)dx \approx \frac{2}{N}\sum_{j=1}^N p(y_j),$$

$$2\int p(x)q(x)dx \approx \frac{2}{N}\sum_{j=1}^N q(x_j).$$

If you cannot simulate from $p$ and $q$, then you may need use some deterministic numerical integration method such as quadrature. Another possibility is to use importance sampling integration, but you need to choose an appropriate importance density function $r$ that you can simulate from as follows:

$$\int p(x)^2dx = \int \frac{p(x)^2}{r(x)}r(x)dx \approx \frac{1}{N}\sum_{j=1}^N \frac{p(z_j)^2}{r(z_j)},$$ where $z_j\sim r$. Analogously for the other quantities. You can also use this trick on $(p-q)^2$.

Anyway, it is also a good practice to compare these estimators with those obtained with quadrature methods, and my main message is that life is not easy, and neither multivariate numerical integration.

$\endgroup$
  • $\begingroup$ Neat answer. Thanks a lot. Do you think it is okay to share samples when estimating each term? $\endgroup$ – Sangwoong Yoon Aug 6 '16 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.