Nowadays the Dirichlet process mixture (DPM) seems to be the default Bayesian approach for density estimation. My question is why not simply use the kernel density estimate (KDE) to model the density? One can surely use a Bayesian sampler (e.g. MCMC) to infer the bandwidth of KDE and thus make it "Bayesian". I would argue that posterior draws from Bayesian KDE are easier to check convergence than DPM. What would be the theoretical benefits of DPM over KDE?

  • 1
    $\begingroup$ Two comments: (1) bandwidth choice for KDE is not a trivial problem. You can set it to zero and have a solution that perfectly fits the data, but is a horrible approximation of the density. (2) Methods for checking the convergence of MCMC for both models would be exactly the same no matter of the model definition. $\endgroup$ – Tim Nov 26 '19 at 21:13

After some research and thinking, here would be my own tentative answer to the question I posted; just in case someone else is interested in this question.

Given n data points, KDE uses a mixture of n kernels to approximate the "true" density while DPM, in finite samples, typically ends up with a smaller mixture even though it theoretically uses an infinite number of mixtures. Moreover, KDE fixes the mixing weights at 1/n while DPM allows it to follow the stick breaking process. Assuming Gaussian kernels, it is easy to see that KDE fixes the means of the kernels at the data points while DPM estimates them from data. Taken together, DPM could argue that it is more parsimonious (using a smaller mixture) and is more flexible (not fixing mixture weights and kernel parameters) than KDE. The empirical findings in http://mlg.eng.cam.ac.uk/pub/pdf/GoeRas10.pdf contains a comparison of the performance of KDE and DPM and seems to suggest that DPM performs better.

(To Tim's comments: Rob Hyndman has a paper on estimating KDE bandwidth https://robjhyndman.com/publications/bandwidth-selection-for-multivariate-kernel-density-estimation-using-mcmc/. There are probably more papers on that topic. As for convergence, depending on the algorithm for DPM, the number of kernels as well as their labels change across MCMC iterations and hence makes it challenging to check the draws' convergence.)

| cite | improve this answer | |
  • $\begingroup$ I'd like to hear more about your exploration of this question--can we discuss by email? I'm here: sethrf.com $\endgroup$ – stackoverflax Dec 20 '19 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.