# Posterior Distribution in a Bayesian Multivariate Normal Model

I am currently working on a Bayesian inference problem and would appreciate some help on computing the posterior distribution of a hyperparameter within a specific multivariate normal model. Below, I provide a simplified outline of my problem and the challenges I'm encountering.

• Data and Distribution: I have a dataset where the observed variable, denoted as $$\mathbf{A}_{N^2 \times 1}$$, follows a multivariate normal distribution: $$\mathcal{N}(\mathbf{A} | (\mathbf{C} \otimes \mathbf{C}) \mathbf{vec}(\mathbf{b}), \sigma_{\mathbf{A}}^2 \mathbb{I})$$. Here, $$\mathbf{C}_{N \times K}$$ is a binary latent variable.
• Latent Variable: The latent variable in this model is $$\mathbf{vec}(\mathbf{b})$$, a vector of size $$K^2 \times 1$$.
• Prior Distribution: The prior over $$\mathbf{b}$$ is Gaussian with zero mean and covariance $$\sigma_{\mathbf{b}}^2 \mathbb{I}$$.
• Objective: My goal is to compute the posterior distribution of the hyperparameter $$\sigma_{\mathbf{b}}^2$$.

I think I must compute the following posterior distribution $$P(\sigma_{\mathbf{b}}^2|\mathbf{A},\mathrm{rest})\propto\int \mathcal{N}\Big(\mathbf{A}|(\mathbf{C}\otimes\mathbf{C})\mathbf{vec}(\mathbf{b}),\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}\Big)\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mathbf{0},\sigma_{\mathbf{b}}^2\mathbb{I}\Big)\;\mathrm{d}\mathbf{b}\times\text{Inv-Gam}(\sigma_{\mathbf{b}}^2|c,d)$$.

If I can integrate out $$\mathbf{b}$$ and use completing the square here I would have $$\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mu_{\mathbf{b}}, \Sigma_{\mathbf{b}}\Big)$$ where $$\Sigma_{\mathbf{b}}=\Big(\frac{1}{\sigma_{\mathbf{b}}^2}\mathbb{I}+\frac{1}{\sigma_{\mathbf{A}}^2}(\mathbf{C}\otimes\mathbf{C})^T(\mathbf{C}\otimes\mathbf{C})\Big)^{-1}$$ and $$\mu_{\mathbf{b}}=\frac{1}{\sigma_{\mathbf{A}}^2}\Sigma_{\mathbf{b}}^{-1}(\mathbf{C}\otimes\mathbf{C})^T\mathbf{A}$$.

### Challenges:

Computational Complexity: The integration involves a complex multivariate normal distribution, which makes it challenging to use the conjugacy properties of the Normal and Inverse Gamma distributions for estimating parameters $$c$$ and $$d$$ in a closed form.

### Specific Question:

Gibbs Sampling: In the context of Gibbs Sampling, are there any strategies or simplifications that could aid in computing this posterior distribution, particularly for obtaining a closed-form distribution for the posterior of $$\sigma_{\mathbf{b}}^2$$? Based on my calculation the posterior should be $$p(\sigma^2_{\mathbf{b}}|\mathbf{A},..)=\mathcal{N}\Big(\mathbf{A}\Big|\;\boldsymbol{0},\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}+\sigma_{\mathbf{b}}^2(\mathbf{C}\otimes\mathbf{C})(\mathbf{C}\otimes\mathbf{C})^T\Big)\times\frac{d^c}{\Gamma(c)(\sigma^2_{\mathbf{b}})^{c+1}}\exp(-\frac{d}{\sigma^2_{\mathbf{b}}})$$ while $$\sigma^2_{\mathbf{b}}$$ appears in the covariance of a multivariate Gaussian which is not intuitive to me how could I extract it in oorder to build the posterior of $$\sigma^2_{\mathbf{b}}$$.

Any solutions, or examples of similar computations would be appreciated.

• In a Gibbs sampler you typically sample from the joint posterior distribution of the model parameters by iteratively sampling from full conditional (posterior) distributions of individual parameters or parameter blocks. Do you want to compute the full conditional distribution of $\sigma_{\mathbf b}^2$? Commented Nov 28, 2023 at 10:35
• yes, I do want to estimate the full conditional posterior distribution. Commented Nov 28, 2023 at 14:08
• why going for IG if you can conveniently assume that $\sigma_b^2I$ has the inverse Wishart distro? Commented Nov 29, 2023 at 9:13

If I understand you correctly, the only distributions depending on $$\sigma^2_{\mathbf{b}}$$ are the prior distributions of $$\sigma^2_{\mathbf{b}}$$ and $$\tilde{\mathbf{b}} \mathrel{:=} \mathrm{vec}(\mathbf{b})$$ given by $$\mathop{\mathcal{IG}}\left(c,d\right)$$ and $$\mathop{\mathcal N_{K^2}}\left(\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right)$$, respectively.
Thus, the full conditional density of $$\sigma^2_{\mathbf{b}}$$ is proportional to \begin{align} &\mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c,d\right) \cdot \mathop{\mathcal N_{K^2}}\left(\tilde{\mathbf{b}};\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right) \\ &\propto \left(\sigma^2_{\mathbf{b}}\right)^{-c -1} \exp\left(-\frac{d}{\sigma^2_{\mathbf{b}}}\right) \cdot \left(\sigma^2_{\mathbf{b}}\right)^{-K^2/2} \exp\left(-\frac{1}{2\sigma^2_{\mathbf{b}}} \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}\right) \\ &= \left(\sigma^2_{\mathbf{b}}\right)^{-c -K^2/2 - 1} \exp\left(-\frac{d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2}{\sigma^2_{\mathbf{b}}}\right)\\ &\propto \mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c + K^2/2, d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2\right). \end{align} Therefore, the full conditional distribution of $$\sigma^2_{\mathbf{b}}$$ is an inverse gamma distribution with shape parameter $$c + K^2/2$$ and scale parameter $$d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2$$.
• @stamerkur I have tried this answer computationally and it is not working. I believe this is not a corrct way to compute the posterior distribution over $\sigma_{\mathbf{b}}^2$. Commented Nov 29, 2023 at 1:07
• @Dalek I'm sorry to hear that this doesn't work for you. The full conditional density of $\sigma^2_{\mathbf{b}}$ is proportional to the joint density of $\left(\mathbf A, \mathbf{C}, \mathbf{b}, \sigma^2_{\mathbf{A}}, \sigma^2_{\mathbf{b}}\right)$ viewed as a function of $\sigma^2_{\mathbf{b}}$. If I understand you correctly, this joint density factorizes as $p(\mathbf A|\mathbf{C}, \mathbf{b}, \sigma^2_{\mathbf{A}}) p(\mathbf{C})p(\mathbf{b}| \sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{A}})$, in which I assumed ... ctd Commented Nov 29, 2023 at 9:38
• ctd ... only $p(\mathbf{b}| \sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{b}})$ depends on $\sigma^2_{\mathbf{b}}$. Does this reflect your model specification? If not, please edit your question accordingly. Commented Nov 29, 2023 at 9:38