Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure?

I am currently evaluating the procedure using three test cases in 1-D, 2-D, and 5-D. I draw $N=10^k$ samples from a known distribution $P$, with $k=1,\ldots,6$, and calculate a pdf estimate $\hat{P}$ using the implemented method. I repeat this process for each example and each value of $k$ for a number $n_{\text{run}}$ of times to average out random sampling effects. What I am missing is a suitable similarity measure.

Candidates I've considered:

  • KL Divergence $D_{KL}(P || \hat{P}) = \int p(x) \log \frac{p(x)}{\hat{p}(x)} \mathrm{d} x$. One problem I encounter with this approach is that there may be regions where my estimate is strictly zero and the original distribution is not. Even if that concerns only a very small part of the data space, this would lead to $D_{KL}(P || \hat{P})=\infty$, which does not seem useful.
  • KL Divergence $D_{KL}(\hat{P} || P) = \int \hat{p}(x) \log \frac{\hat{p}(x)}{p(x)} \mathrm{d} x$. By changing the order of the arguments, I could get rid of the "zero-problem" mentioned above. Maybe this might be a good candidate? The KL divergence is not a useful distance measure in all cases, though.
  • Wasserstein distance. This sounds very nice (see the link above), but the general multivariate case seems to be highly nontrivial to implement. (Please correct me if I'm wrong.)
  • Jensen-Shannon divergence. A symmetrized and smoothed version of the KL divergence, which is guaranteed to always be finite, and can also be implemented in N-D. This seems like a good option to me. (Wiki)
  • Mean squared error $\mathrm{MSE} = \frac{1}{A}\int (p(x)-\hat{p}(x))^2 \mathrm{d}x$. This is, of course, a standard performance measure in many estimation settings. One problem I see here is that the value will be highly dependend upon the "peakiness" of $P$: if there is a large peak, the MSE will be large, whereas if $P$ is rather uniform, the MSE will be small.

Right now, I would probably go with the JS divergence. Would that be a good choice? Are there any other good (if not better) options I should be aware of?

  • $\begingroup$ You don't supply the information needed to make recommendations. What do you mean by "good choice"? What properties of this density estimator matter to you? $\endgroup$
    – whuber
    Aug 20, 2021 at 17:03
  • $\begingroup$ @whuber That is part of the question TBH. What kind of performance analysis would you want to see in a paper to be convinced that a method is generally useful as a density estimation scheme? What I was trying to do so far was to show that the accuracy gets better with increasing values of N, because that seemed like a useful and important property to me. $\endgroup$
    – Eike P.
    Aug 20, 2021 at 17:23
  • $\begingroup$ The background is that I needed an efficient density estimation scheme that works in N>3 dimensions for an application I'm working on. Now I essentially just want to show (and verify!) that the method I designed and implemented "works". $\endgroup$
    – Eike P.
    Aug 20, 2021 at 17:26
  • 1
    $\begingroup$ I'm afraid that doesn't help us much. For instance, if the true distribution is bounded but the estimate goes beyond the bounds, how big a problem would that be? What about if the estimator produces a discontinuous density and your application can't handle that, even if the estimator is extremely accurate? Etc., etc. Ultimately, the application's sensitivity to estimation errors is what need to drive your assessment of the estimator. $\endgroup$
    – whuber
    Aug 20, 2021 at 20:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.