3
$\begingroup$

Suppose we have $p$ dimensional vectors $Y_i$ which we model with $f_Y (y |\theta) = \sum \pi_k N(y | \mu_k, \Sigma_k)$ with $\theta$ being a catch all for the model parameters (the number of components might be a finite known/unknown number or infinite as in Dirichlet process mixtures). The prior on $\pi$ will either be a uniform or Stick-breaking prior with perhaps a hyperprior on the associated precision parameter. What are good default prior choices for the cluster components $(\mu_k, \Sigma_k)$? I have been using a conjugate Normal-inverse-Wishart, i.e. $$ (\mu_k, \Sigma_k) \sim \mathcal N(\mu_k |m, \Sigma_k / n_0) \mathcal W^{-1} (\Sigma_k | \Psi, \nu). $$ I then fix $\nu, n_0$ relatively small and either estimate $m$ and $\Psi$ empircally, or specify independent hyperpriors and estimate the key parameters of the hyperpriors empircally (from what I can tell, there is some evidence that people do this since it seems to be what is done in the vignettes for DPpackage).

Ultimately, I'd like to set things up so that the prior should be relatively uninformative (with at least some hope of adding prior knowledge in a systematic way), but there are a lot of parameters floating around and the individual influence of each one isn't always transparent. I've come across some papers on this issue that give guidance for choosing parameters empirically but they mainly focus on $p = 1$. Given that these models are pervasive in Bayesian nonparametrics, I figure guidelines must exist and that I just haven't found them.

Ideally, I'd like answers here to be specific as possible; in particular, I'm most interested in choosing the hyperparameters in the priors/hyperpriors.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ The Normal-inverse-Wishart seems to be a good choice. If you want to use vague priors, you might consider using a multivariate Cauchy or $t$ instead of a normal which is more or less common . Not an elegant solution, just some random thoughts. The main issue is to come up with heavy-tailed distributions on positive-definite matrices. $\endgroup$
    – user10525
    Commented Jul 26, 2012 at 16:27
  • $\begingroup$ @Procrastinator the Wishart does seem to be the most suspect component, it's just very difficult for me to get away from conjugacy since I'm mainly using Dirichlet Processes which WinBUGS seems to fail at fitting with reasonable answers and it's hard to justify the time it takes to code samplers for non-conjugate priors. I'm much more troubled by choosing hyperparameters though. $\endgroup$
    – guy
    Commented Jul 26, 2012 at 16:57
  • $\begingroup$ Off-topic: Just out of curiosity, why do you think WinBUGS fails at fitting Dirichlet processes? I use them more-or-less regularly for parts of models (w/ JAGS), and with a little care (choosing a prior on the concentration parameter that gives me roughly the right prior expectation for the number of clusters and s.d. of same) I have had ... reasonable ... results. I also use the zeroes / ones trick for samplers, with, usually, decent mixing and convergence results I must say. (Should probably put this sub-thread in chat.) $\endgroup$
    – jbowman
    Commented Jul 26, 2012 at 17:07
  • $\begingroup$ @jbowman It may be particular to my problem but I've had it where a one-cluster area of the posterior is very difficult to leave even when the data is very clearly not well described by one cluster (intuitively the one cluster area is locally the best explanation of the data in the posterior). I suspect it's a mixing issue since I use the same model with my own optimized code and it mixes very well, but it very well could be an error on my part. I know people use it (WinBUGS), apparently with success. $\endgroup$
    – guy
    Commented Jul 26, 2012 at 18:26
  • 4
    $\begingroup$ The normal-Wishart choice is indeed the default choice for mixtures as it gives you closed-form formulas and a straightforward Gibbs sampler. There is no non-informative version as you need the prior to be proper. For the choices of hyperparameters, I would pick $m$ as a central value of the dataset, $\Psi$ associated with the scale of the dataset, $n_0=1/n$ and $\nu$ equal to $3$ to $5$ to ensure existing moments. $\endgroup$
    – Xi'an
    Commented Jul 26, 2012 at 20:01

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.