2
$\begingroup$

I am looking to choose a prior that helps me avoid singularities (as mentioned in this answer) in the covariance matrices of a GMM model. The Jeffrey prior (or a simple improper prior) would be very convenient but since its form is $p(\Sigma) \propto |\Sigma|^{-k}$ I'm not sure if it prevents them or not, because it is this term (which equals $\sigma^{-1}$ in the univariate case) that causes the problem in the first place according to the answer mentioned above. Would only proper priors do the trick?

$\endgroup$
3
  • 1
    $\begingroup$ For covariance matrices you need a prior with support over positive-definite matrices. The Inverse-Wishart distribution is that prior. $\endgroup$ – kedarps May 17 '18 at 14:59
  • 1
    $\begingroup$ @kedarps I think you should make that into an answer, because it is :) $\endgroup$ – conjectures May 18 '18 at 12:27
  • 3
    $\begingroup$ The Stan Prior Choice Recommendation wiki (github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations) lists several options. I'm not an expert but I would recommend splitting the covariance matrix into a correlation $C$ matrix and a diagonal matrix of standard deviations $S$; then $SCS$ is your covariance matrix and you can specify priors separately for each. I hear the LKJ prior is useful for correlation matrices. I hope this helps! $\endgroup$ – Maurits M May 18 '18 at 13:50
1
$\begingroup$

The singularity issue that is mentioned in the answer is a problem with the MLE estimation for the GMM, when estimated using the expectation-maximization (EM) algorithm. If you use a Bayesian approach, you will not face singularity issues.

Typically, for covariance matrices you need a prior with support over positive-definite matrices which can be achieved by using the Inverse-Wishart prior.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.