1
$\begingroup$

I have seen density estimation methods which are pretty old. Specifically, I am referring to Parzen Window method. When I read the original Parzen's paper, I was amazed by it's beauty and I know that Parzen window method is still pretty common. For instance, it is used by Ian Goodfellow in his original GAN paper. Are there more advanced density estimation methods and what are their pros and cons? Is density estimation as active field of research? If so, what are some recent influential papers in this area?

$\endgroup$
  • $\begingroup$ if you're willing to view generative models as performing density estimation, then yes, plenty. $\endgroup$ – shimao Jul 21 at 18:53
  • 2
    $\begingroup$ I have found Bernard Silverman's publications on density estimation to be clearly written and useful. $\endgroup$ – BruceET Jul 21 at 22:15
3
$\begingroup$

There is plenty of modern work going on in density estimation. For instance, the ICLR 2020 conference submission deadline was in the last 24 hours; all the submissions are here, and I'm sure there are at least one or two that belong on this list but I just haven't seen them yet.

KDE is still a workhorse in low dimensions, but its application to settings like the one in the original GAN paper was seriously flawed and it is (thankfully) no longer really done in those settings. Yet even on KDE there's still work going on, e.g. this paper from a few years ago about consistency of KDE with fixed bandwidths, or algorithmically speaking this paper that could be viewed sort of as using a "deep kernel" instead of something like a Gaussian.

Some lines of recent work on density estimation include:

  • Autoregressive models: here we factor $p(x_1, x_2, x_3) = p(x_1) p(x_2 \mid x_1) p(x_3 \mid x_1, x_2)$, and estimate each density with some parametric model. Influential papers include RNADE, PixelRNN, PixelCNN, WaveNet, and MADE. This models often produce both good densities and good samples, but can be slow, since they need to go through e.g. a whole image pixel-by-pixel.

  • Normalizing flows: these models start with a simple distribution, say a Gaussian, and stack a series of invertible transformations on top of that which are constrained to have simple Jacobians, so that you can compute densities with the change of variables formula. Influential papers include the original, RealNVP, GLOW, and FFJORD. (MAF is a cool combination of MADE with flows.) These models also produce both decent likelihoods and samples, but because each transformation needs to be invertible and have a simple Jacobian-determinant, they typically need to be fairly deep and the latent factors need to be the full size of the output, so they tend to be quite memory-intensive.

  • There's also a fair amount of work on fitting unnormalized density functions with methods other than maximum likelihood, with techniques like score matching and noise contrastive estimation; some recent examples of score matching include a recent paper of mine and a more recent technique you should maybe use instead.

  • There's also a huge recent literature on generative models that eschew explicit density functions entirely, like GANs and their many variants (including a few of mine). Variational autoencoders are a halfway point there, in that they use a bound on likelihoods but don't really estimate densities.

$\endgroup$
2
$\begingroup$

I agree with BruceET that Silverman's work offers a good read on kernel density estimation (KDE). However, it is far from recent.

A good recent overview on KDE is given in "Scott - Multivariate Density Estimation". You can also have a look at "Simonoff - Smoothing methods in statistics" (less recent). One major development after Parzen's and Silverman's work is variable (or adaptive) kernel density estimation (VKDE), where you use a different bandwidths $h_j$ for each sample point $x_j$: $$ \hat f(x) = \frac{1}{J}\sum_{j=1}^{J} h_j^{-d} K\left(\frac{x-x_j}{h_j}\right), $$ (this approach can be generalized by using unisotropic kernels, i.e. $h_j$ being a matrix). VKDE is also discussed in Scott's book with many references to Abramson, Jones, Terrell, Wand and others. One particular publication I would stress (this is subjective) is the one of Botev, Grotowski and Kroese, for which there is a free Matlab implementation by Botev here.

Remark: It is not quite accurate that VKDE began after Silverman's work, since it was pioneered in 1977 by "Breiman, Meisel, Purcell - Variable kernel estimates of multivariate densities". But a lot of its development came later.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.