Best way to evaluate PDF estimation methods

Question

I wish to test some of my ideas that I think are better than anything that I have seen. I could be wrong but I'd like to test my ideas and vanquish my doubts by more certain observations.

What I have been thinking to do is the following:

1. Analytically define a set of distributions. Some of these are easy ones like Gaussian, uniform, or Tophat. But some of these must be difficult and challenging such as the Simpsons distribution.
2. Implement software based on those analytical distributions, and use them to generate some samples.
3. Because the distributions are analytically defined, I already -by definition- know their true PDFs. This is great.
4. Then I'll test the following PDF estimation methods against the samples above:
   • Existing PDF estimation methods (like KDE with various kernels and bandwidths).
   • My own idea that I think is worth trying.
5. Then I will measure the error of the estimations against the true PDFs.
6. Then I will better know which of the PDF estimation methods is good.

My questions are:

• Q1: Are there any improvements over my plan above?
• Q2: I find it difficult for me to analytically define many true PDFs. Is there already a comprehensive list of many analytically defined true PDFs with varying difficulties (including very difficult ones) that I can re-use here?

Answer 1

A2: You could test your methods in 1D on the following set of benchmarks.

Answer 2

• A1. This sounds like a sensible plan to me. Just to mention a couple of points. You'll want to test with different error metrics ($L^p$, K-L divergence, etc.) since methods will perform differently depending on the loss function. Also, you'll want to test for different number of samples. Finally, many density estimation methods perform notoriously badly near discontinuities/boundaries, so be sure to include truncated pdfs in your set.

• A2. Are you interested only in 1-D pdfs or is your plan to test the multivariate case? As for a benchmark suite of pdfs, I asked a somewhat related question in the past with the goal of testing MCMC algorithms, but I did not find anything like a well-established set of pdfs.

If you have plenty of time and computational resources, you might consider performing some sort of adversarial testing of your idea:

• Define a very flexible parametric family of pdfs (e.g., a large mixture of a number of known pdfs), and move around the parameter space of the mixture via some nonconvex global optimization method (*) so as to minimize performance of your method and maximize performance of some other state-of-the-art density estimation method (and possibly vice versa). This will be a strong test of the strength/weakness of your method.

Finally, the requirement of being better than all other methods is an excessively high bar; there must be some no free lunch principle at work (any algorithm has some underlying prior assumption, such as smoothness, length scale, etc.). In order for your method to be a valuable contribution, you only need to show that there are regimes/domains of some general interest in which your algorithm works better (the adversarial test above can help you find/define such a domain).

(*) Since your performance metric is stochastic (you will be evaluating it via Monte Carlo sampling), you may also want to check this answer about optimization of noisy, costly objective functions.

Answer 3

Q1: Are there any improvements over my plan above?

That depends. Mixture distribution residuals often result from doing silly things like specifying an unnecessary mixture distribution as a data model to begin with. So, my own experience suggests to at least specify as many mixture distribution terms in the output as there are in the model. Moreover, the mixture PDF's output are unlike the PDF's in the model. The Mathematica default search includes mixture distributions with two terms, and can be specified as a larger number.

Q2: Is there already a comprehensive list of many analytically defined true PDFs with varying difficulties (including very difficult ones) that I can re-use here?

This is a list from Mathematica's FindDistribution routine:

Possible continuous distributions for TargetFunctions are: BetaDistribution, CauchyDistribution, ChiDistribution, ChiSquareDistribution, ExponentialDistribution, ExtremeValueDistribution, FrechetDistribution, GammaDistribution, GumbelDistribution, HalfNormalDistribution, InverseGaussianDistribution, LaplaceDistribution, LevyDistribution, LogisticDistribution, LogNormalDistribution, MaxwellDistribution, NormalDistribution, ParetoDistribution, RayleighDistribution, StudentTDistribution, UniformDistribution, WeibullDistribution, HistogramDistribution.

Possible discrete distributions for TargetFunctions are: BenfordDistribution, BinomialDistribution, BorelTannerDistribution, DiscreteUniformDistribution, GeometricDistribution, LogSeriesDistribution, NegativeBinomialDistribution, PascalDistribution, PoissonDistribution, WaringYuleDistribution, ZipfDistribution, HistogramDistribution, EmpiricalDistribution.

The internal information criterion uses a Bayesian information criterion together with priors over TargetFunctions.