Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure?
I am currently evaluating the procedure using three test cases in 1-D, 2-D, and 5-D. I draw $N=10^k$ samples from a known distribution $P$, with $k=1,\ldots,6$, and calculate a pdf estimate $\hat{P}$ using the implemented method. I repeat this process for each example and each value of $k$ for a number $n_{\text{run}}$ of times to average out random sampling effects. What I am missing is a suitable similarity measure.
Candidates I've considered:
- KL Divergence $D_{KL}(P || \hat{P}) = \int p(x) \log \frac{p(x)}{\hat{p}(x)} \mathrm{d} x$. One problem I encounter with this approach is that there may be regions where my estimate is strictly zero and the original distribution is not. Even if that concerns only a very small part of the data space, this would lead to $D_{KL}(P || \hat{P})=\infty$, which does not seem useful.
- KL Divergence $D_{KL}(\hat{P} || P) = \int \hat{p}(x) \log \frac{\hat{p}(x)}{p(x)} \mathrm{d} x$. By changing the order of the arguments, I could get rid of the "zero-problem" mentioned above. Maybe this might be a good candidate? The KL divergence is not a useful distance measure in all cases, though.
- Wasserstein distance. This sounds very nice (see the link above), but the general multivariate case seems to be highly nontrivial to implement. (Please correct me if I'm wrong.)
- Jensen-Shannon divergence. A symmetrized and smoothed version of the KL divergence, which is guaranteed to always be finite, and can also be implemented in N-D. This seems like a good option to me. (Wiki)
- Mean squared error $\mathrm{MSE} = \frac{1}{A}\int (p(x)-\hat{p}(x))^2 \mathrm{d}x$. This is, of course, a standard performance measure in many estimation settings. One problem I see here is that the value will be highly dependend upon the "peakiness" of $P$: if there is a large peak, the MSE will be large, whereas if $P$ is rather uniform, the MSE will be small.
Right now, I would probably go with the JS divergence. Would that be a good choice? Are there any other good (if not better) options I should be aware of?
