I am currently working on a Bayesian inference problem and would appreciate some help on computing the posterior distribution of a hyperparameter within a specific multivariate normal model. Below, I provide a simplified outline of my problem and the challenges I'm encountering.

  • Data and Distribution: I have a dataset where the observed variable, denoted as $\mathbf{A}_{N^2 \times 1}$, follows a multivariate normal distribution: $\mathcal{N}(\mathbf{A} | (\mathbf{C} \otimes \mathbf{C}) \mathbf{vec}(\mathbf{b}), \sigma_{\mathbf{A}}^2 \mathbb{I})$. Here, $\mathbf{C}_{N \times K}$ is a binary latent variable.
  • Latent Variable: The latent variable in this model is $\mathbf{vec}(\mathbf{b})$, a vector of size $K^2 \times 1$.
  • Prior Distribution: The prior over $\mathbf{b}$ is Gaussian with zero mean and covariance $\sigma_{\mathbf{b}}^2 \mathbb{I}$.
  • Objective: My goal is to compute the posterior distribution of the hyperparameter $\sigma_{\mathbf{b}}^2$.

I think I must compute the following posterior distribution $P(\sigma_{\mathbf{b}}^2|\mathbf{A},\mathrm{rest})\propto\int \mathcal{N}\Big(\mathbf{A}|(\mathbf{C}\otimes\mathbf{C})\mathbf{vec}(\mathbf{b}),\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}\Big)\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mathbf{0},\sigma_{\mathbf{b}}^2\mathbb{I}\Big)\;\mathrm{d}\mathbf{b}\times\text{Inv-Gam}(\sigma_{\mathbf{b}}^2|c,d)$.

If I can integrate out $\mathbf{b}$ and use completing the square here I would have $\mathcal{N}\Big(\mathbf{vec}(\mathbf{b})|\mu_{\mathbf{b}}, \Sigma_{\mathbf{b}}\Big)$ where $\Sigma_{\mathbf{b}}=\Big(\frac{1}{\sigma_{\mathbf{b}}^2}\mathbb{I}+\frac{1}{\sigma_{\mathbf{A}}^2}(\mathbf{C}\otimes\mathbf{C})^T(\mathbf{C}\otimes\mathbf{C})\Big)^{-1}$ and $\mu_{\mathbf{b}}=\frac{1}{\sigma_{\mathbf{A}}^2}\Sigma_{\mathbf{b}}^{-1}(\mathbf{C}\otimes\mathbf{C})^T\mathbf{A}$.


Computational Complexity: The integration involves a complex multivariate normal distribution, which makes it challenging to use the conjugacy properties of the Normal and Inverse Gamma distributions for estimating parameters $c$ and $d$ in a closed form.

Specific Question:

Gibbs Sampling: In the context of Gibbs Sampling, are there any strategies or simplifications that could aid in computing this posterior distribution, particularly for obtaining a closed-form distribution for the posterior of $\sigma_{\mathbf{b}}^2$? Based on my calculation the posterior should be $ p(\sigma^2_{\mathbf{b}}|\mathbf{A},..)=\mathcal{N}\Big(\mathbf{A}\Big|\;\boldsymbol{0},\sigma_{\mathbf{A}}^2\mathbb{I}_{N^2}+\sigma_{\mathbf{b}}^2(\mathbf{C}\otimes\mathbf{C})(\mathbf{C}\otimes\mathbf{C})^T\Big)\times\frac{d^c}{\Gamma(c)(\sigma^2_{\mathbf{b}})^{c+1}}\exp(-\frac{d}{\sigma^2_{\mathbf{b}}})$ while $\sigma^2_{\mathbf{b}}$ appears in the covariance of a multivariate Gaussian which is not intuitive to me how could I extract it in oorder to build the posterior of $\sigma^2_{\mathbf{b}}$.

Any solutions, or examples of similar computations would be appreciated.

  • $\begingroup$ In a Gibbs sampler you typically sample from the joint posterior distribution of the model parameters by iteratively sampling from full conditional (posterior) distributions of individual parameters or parameter blocks. Do you want to compute the full conditional distribution of $\sigma_{\mathbf b}^2$? $\endgroup$
    – statmerkur
    Commented Nov 28, 2023 at 10:35
  • $\begingroup$ yes, I do want to estimate the full conditional posterior distribution. $\endgroup$
    – Dalek
    Commented Nov 28, 2023 at 14:08
  • $\begingroup$ why going for IG if you can conveniently assume that $\sigma_b^2I$ has the inverse Wishart distro? $\endgroup$
    – Spätzle
    Commented Nov 29, 2023 at 9:13

1 Answer 1


If I understand you correctly, the only distributions depending on $\sigma^2_{\mathbf{b}}$ are the prior distributions of $\sigma^2_{\mathbf{b}}$ and $\tilde{\mathbf{b}} \mathrel{:=} \mathrm{vec}(\mathbf{b})$ given by $\mathop{\mathcal{IG}}\left(c,d\right)$ and $\mathop{\mathcal N_{K^2}}\left(\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right) $, respectively.

Thus, the full conditional density of $\sigma^2_{\mathbf{b}}$ is proportional to \begin{align} &\mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c,d\right) \cdot \mathop{\mathcal N_{K^2}}\left(\tilde{\mathbf{b}};\mathbf 0, \sigma_{\mathbf{b}}^2 \mathbf I\right) \\ &\propto \left(\sigma^2_{\mathbf{b}}\right)^{-c -1} \exp\left(-\frac{d}{\sigma^2_{\mathbf{b}}}\right) \cdot \left(\sigma^2_{\mathbf{b}}\right)^{-K^2/2} \exp\left(-\frac{1}{2\sigma^2_{\mathbf{b}}} \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}\right) \\ &= \left(\sigma^2_{\mathbf{b}}\right)^{-c -K^2/2 - 1} \exp\left(-\frac{d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2}{\sigma^2_{\mathbf{b}}}\right)\\ &\propto \mathop{\mathcal{IG}}\left(\sigma^2_{\mathbf{b}};c + K^2/2, d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2\right). \end{align} Therefore, the full conditional distribution of $\sigma^2_{\mathbf{b}}$ is an inverse gamma distribution with shape parameter $c + K^2/2$ and scale parameter $d + \tilde{\mathbf{b}}^\mathsf{T} \tilde{\mathbf{b}}/2$.

  • $\begingroup$ @stamerkur I have tried this answer computationally and it is not working. I believe this is not a corrct way to compute the posterior distribution over $\sigma_{\mathbf{b}}^2$. $\endgroup$
    – Dalek
    Commented Nov 29, 2023 at 1:07
  • $\begingroup$ @Dalek I'm sorry to hear that this doesn't work for you. The full conditional density of $\sigma^2_{\mathbf{b}}$ is proportional to the joint density of $\left(\mathbf A, \mathbf{C}, \mathbf{b}, \sigma^2_{\mathbf{A}}, \sigma^2_{\mathbf{b}}\right)$ viewed as a function of $\sigma^2_{\mathbf{b}}$. If I understand you correctly, this joint density factorizes as $p(\mathbf A|\mathbf{C}, \mathbf{b}, \sigma^2_{\mathbf{A}}) p(\mathbf{C})p(\mathbf{b}| \sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{A}})$, in which I assumed ... ctd $\endgroup$
    – statmerkur
    Commented Nov 29, 2023 at 9:38
  • $\begingroup$ ctd ... only $p(\mathbf{b}| \sigma^2_{\mathbf{b}})p(\sigma^2_{\mathbf{b}})$ depends on $\sigma^2_{\mathbf{b}}$. Does this reflect your model specification? If not, please edit your question accordingly. $\endgroup$
    – statmerkur
    Commented Nov 29, 2023 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.