Posterior Distribution in a Bayesian Multivariate Normal Model

Question

I am currently working on a Bayesian inference problem and would appreciate some help on computing the posterior distribution of a hyperparameter within a specific multivariate normal model. Below, I provide a simplified outline of my problem and the challenges I'm encountering.

• Data and Distribution: I have a dataset where the observed variable, denoted as $\mathbf{A}_{N^2 \times 1}$, follows a multivariate normal distribution: $\mathcal{N}(\mathbf{A} | (\mathbf{C} \otimes \mathbf{C}) \mathbf{vec}(\mathbf{b}), \sigma_{\mathbf{A}}^2 \mathbb{I})$. Here, $\mathbf{C}_{N \times K}$ is a binary latent variable.
• Latent Variable: The latent variable in this model is $\mathbf{vec}(\mathbf{b})$, a vector of size $K^2 \times 1$.
• Prior Distribution: The prior over $\mathbf{b}$ is Gaussian with zero mean and covariance $\sigma_{\mathbf{b}}^2 \mathbb{I}$.
• Objective: My goal is to compute the posterior distribution of the hyperparameter $\sigma_{\mathbf{b}}^2$.

I think I must compute the following posterior distribution $P(\sigma_{\mathbf{b}}^2|\mathbf{A},\mathrm{rest})\propto\int \mathcal{N}\Big(\mathbf{A}|(\mathbf{C}\otimes\mathbf{C})\mathbf{vec}(\mathbf{b}),\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}\Big)\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mathbf{0},\sigma_{\mathbf{b}}^2\mathbb{I}\Big)\;\mathrm{d}\mathbf{b}\times\text{Inv-Gam}(\sigma_{\mathbf{b}}^2|c,d)$.

If I can integrate out $\mathbf{b}$ and use completing the square here I would have $\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mu_{\mathbf{b}}, \Sigma_{\mathbf{b}}\Big)$ where $\Sigma_{\mathbf{b}}=\Big(\frac{1}{\sigma_{\mathbf{b}}^2}\mathbb{I}+\frac{1}{\sigma_{\mathbf{A}}^2}(\mathbf{C}\otimes\mathbf{C})^T(\mathbf{C}\otimes\mathbf{C})\Big)^{-1}$ and $\mu_{\mathbf{b}}=\frac{1}{\sigma_{\mathbf{A}}^2}\Sigma_{\mathbf{b}}^{-1}(\mathbf{C}\otimes\mathbf{C})^T\mathbf{A}$.

Challenges:

Computational Complexity: The integration involves a complex multivariate normal distribution, which makes it challenging to use the conjugacy properties of the Normal and Inverse Gamma distributions for estimating parameters $c$ and $d$ in a closed form.

Specific Question:

Gibbs Sampling: In the context of Gibbs Sampling, are there any strategies or simplifications that could aid in computing this posterior distribution, particularly for obtaining a closed-form distribution for the posterior of $\sigma_{\mathbf{b}}^2$? Based on my calculation the posterior should be $ p(\sigma^2_{\mathbf{b}}|\mathbf{A},..)=\mathcal{N}\Big(\mathbf{A}\Big|\;\boldsymbol{0},\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}+\sigma_{\mathbf{b}}^2(\mathbf{C}\otimes\mathbf{C})(\mathbf{C}\otimes\mathbf{C})^T\Big)\times\frac{d^c}{\Gamma(c)(\sigma^2_{\mathbf{b}})^{c+1}}\exp(-\frac{d}{\sigma^2_{\mathbf{b}}})$ while $\sigma^2_{\mathbf{b}}$ appears in the covariance of a multivariate Gaussian which is not intuitive to me how could I extract it in oorder to build the posterior of $\sigma^2_{\mathbf{b}}$.

Any solutions, or examples of similar computations would be appreciated.

Answer 1

If I understand you correctly, the only distributions depending on $\sigma^2_{\mathbf{b}}$ are the prior distributions of $\sigma^2_{\mathbf{b}}$ and $\tilde{\mathbf{b}} \mathrel{:=} \mathrm{vec}(\mathbf{b})$ given by $\mathop{\mathcal{IG}}\left(c,d\right)$ and $\mathop{\mathcal N_{K^2}}\left(\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right) $, respectively.

Thus, the full conditional density of $\sigma^2_{\mathbf{b}}$ is proportional to \begin{align} &\mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c,d\right) \cdot \mathop{\mathcal N_{K^2}}\left(\tilde{\mathbf{b}};\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right) \\ &\propto \left(\sigma^2_{\mathbf{b}}\right)^{-c -1} \exp\left(-\frac{d}{\sigma^2_{\mathbf{b}}}\right) \cdot \left(\sigma^2_{\mathbf{b}}\right)^{-K^2/2} \exp\left(-\frac{1}{2\sigma^2_{\mathbf{b}}} \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}\right) \\ &= \left(\sigma^2_{\mathbf{b}}\right)^{-c -K^2/2 - 1} \exp\left(-\frac{d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2}{\sigma^2_{\mathbf{b}}}\right)\\ &\propto \mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c + K^2/2, d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2\right). \end{align} Therefore, the full conditional distribution of $\sigma^2_{\mathbf{b}}$ is an inverse gamma distribution with shape parameter $c + K^2/2$ and scale parameter $d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2$.