8
$\begingroup$

It seems when I glance around here at the fashionable learning algorithms, things like neural networks, boosted trees, support vector machines, random forests, and friends are promoted for supervised learning problems. Dirichlet processes and their ilk seem to be mentioned mostly in unsupervised learning problems, such as document or image clustering. I do see them get used for regression problems, or as general purpose priors when one wants to do Bayesian statistics in a nonparametric or semiparametric way (e.g. as a flexible prior on the distribution of random effects in certain models) but my limited experience suggests that this doesn't come as much from the machine learning crowd as it does from more traditional statisticians. I've done a small amount of googling on this and I've found a few definitive uses in machine learning of DPs for supervised learning, but it seems like whenever I need to look up something about (say) hierarchical DPs, whatever paper I find the answer in is using it for unsupervised learning.

So, are Dirichlet processes and their cousins most effective as priors for flexible clustering models? Are they not competitive with boosting, SVMs, and neural networks for supervised learning problems? Are they useful only in certain situations for these problems? Or is my general impression incorrect?

$\endgroup$
3
  • $\begingroup$ What sort of regression do mean? in the base distribution of the DP? or in the mixing parameter? I would think you'd have a hard time fitting such a model. $\endgroup$ – probabilityislogic Aug 14 '12 at 7:39
  • $\begingroup$ Or do you mean some sort of generic "regression" where you fit a multivariate DP to the marginal and joint distributions. $\endgroup$ – probabilityislogic Aug 14 '12 at 7:43
  • $\begingroup$ @probabilityislogic fit distribution drawn from DP to joint and then go get the conditionals is the sort of thing I had in mind, with variations on that theme. Modeling the weights in the stick breaking construction is similar. $\endgroup$ – guy Aug 14 '12 at 13:52
1
$\begingroup$

This question isn't getting too much attention, so I'm going to answer to update on what I've found and (hopefully) stimulate discussion. I've ran into an article I'm looking forward to reading that uses DPMs to do classification (Shahbaba and Neal, 2007) which they tested on protein fold data. Essentially it appears that they used did something similar to what I suggested in the comments above. It compared favorable against both neural networks and support vector machines. This comes as a bit of a relief to me since I've dumped a lot of time into these models with an eye towards supervised machine learning problems so it appears I (perhaps) haven't been wasting my time.

$\endgroup$
1
$\begingroup$

Take a look at the DPpackage of R. Dirichlet process can be used at least as a prior for a random effect, and to construct a nonparametric error distribution for regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.