I have a model represented in graphical model and part of this model states that there are N data points that are generated from multivariate normal with shared mean vector and covariance/precision matrix.

$$\theta_n \sim MVN(\mu_{[K]},\Lambda^{-1}_{[K\times K]}) \ \ \ \ \ for \ \ n\in \{1...n\}$$

Where brackets show the dimensions of the vectors and matrices

I wanted to implement a variational Bayes(not variational EM) method that I can use by defining an extra hierarchy where I define priors for the mean vector and precision matrix with Normal and Wishart distribution. Could you help me with how to specify the variational mean field distribution, and how they factorize? Do I need to assume any specific structure to have identification?

There are two ways I can define the priors either independent Normal and Wishart so that

$$p(\mu, \Lambda)=p_{Normal}(\mu)\times p_{Wishart}(\Lambda)$$ or $$p(\mu, \Lambda)=p_{Normal}(\mu|\Lambda)\times p_{Wishart}(\Lambda)$$

I would rather go with a simpler one where the two are independent. In this case, let's say I define the $\mu\sim MVN(\mu_0, S^{-1})$, and $\Lambda\sim \mathcal{W}(W,\nu)$. For the variational distribution that factorizes I specify the one for $\theta$ as below so that: $$q(\theta_{n})=MVN(\gamma_{n},\lambda\mathbb{I}_{K})$$ I have two questions, one how do I define the variational distribution for the $\mu$ and $\Lambda$ and how do I set the parameters in both the data generating process, and in the variational distribution. Should I expect any identification issues?

This is in a way related to the correlated topic models where there is an additional hierarchy used above the mean and covariance.

  • $\begingroup$ Here is the link to the paper that uses VI to estimate parameters of multivariate Gaussion. VI for Multi Gaussion. (see Algorithm 1) $\endgroup$ – chandresh May 22 '18 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.