Skip to main content
1 of 4
moreblue
  • 1.6k
  • 1
  • 10
  • 26

Your try

Here the problem setting is follows:

$$ \begin{aligned} \mathbf{y}_1, \cdots, \mathbf{y}_n | \Sigma &\sim \mathcal{N}\left( \theta, \Sigma \right) \\[10pt] \Sigma &\sim \mathcal{W}^{-1} \left( \nu_0, \mathbf{S}_0 \right) \end{aligned} $$

where $\mathcal{W}^{-1}(\nu, \mathcal{\Psi})$ denotes the inverse-Wishart distribution, which is a generalization of inverse gamma$(\alpha, \beta)$ distribution when dimension $p=1$, with $\nu = 2\alpha, \mathcal{\Psi} = 2\beta$.

Note that we have assumed a given (nonrandom) $\theta$. This is why we can write

$$ \mathbf{y}_1, \cdots, \mathbf{y}_n | \Sigma \sim \mathcal{N}\left( \theta, \Sigma \right) $$

in the first place, rather than $\mathbf{y}_1, \cdots, \mathbf{y}_n | \Sigma, \theta \sim \mathcal{N}\left( \theta, \Sigma \right)$.


Posterior when both $\theta, \Sigma$ : unknown

Alternatively, let us suppose $\theta$ is random, so that we can assume a prior on $\theta$. This is exactly where your idea may kick in. Note that when we start assuming

$$ \begin{aligned} \mathbf{y}_1, \cdots, \mathbf{y}_n | \Sigma, \theta &\sim \mathcal{N}\left( \theta, \Sigma \right) \\[10pt] \Sigma &\sim \mathcal{W}^{-1} (\nu_0, \mathbf{S}_0) \end{aligned} $$

we also simply get the posterior of $\Sigma | \vec{\mathbf{y}}, \theta$ , since by the Bayes rule,

$$ \begin{aligned} p(\Sigma | \vec{\mathbf{y}}, \theta) &= \frac{p(\vec{\mathbf{y}}|\Sigma, \theta) p(\Sigma | \theta)}{\int_{\tilde{\Sigma}} p(\vec{\mathbf{y}}|\tilde{\Sigma}, \theta) p(\tilde{\Sigma}| \theta)} \\[8pt] &\propto \mathcal{N} \left( \nu_0 + n, [\mathbf{S} + \mathbf{S}_\theta]^{-1} \right) \end{aligned} $$

as in the equation (7.9). Note that unlike the above, we cannot say $\Sigma|\theta \equiv \Sigma$. This is exactly why your idea might need caveats. Rather, if we allow an additional information to be

$$ \theta | \Sigma \sim \mathcal{N} (\theta_0, \Sigma/m) $$

for a nonrandom hyperparameters $\theta_0$,$m$, things get simpler, i.e. we can get a closed-form solution.


Posterior when both $\theta | \Sigma$ : normal

The joint density of $\theta$, $\Sigma$ is commonly referred to as Normal-inverse-Wishart distribution, which is simply formulated as

$$ \theta, \Sigma \sim \mathcal{N} (\theta_0, \Sigma/m) \mathcal{W}^{-1} (\nu_0, \mathbf{S}_0) \equiv NIW(\theta, m, \mathbf{S}_0, \nu_0) $$

It can be shown that the posterior distribution is also the Normal-inverse-Wishart, as

$$ \begin{aligned} \theta | \Sigma , \vec{\mathbf{y}} &\sim \mathcal{N} (\theta_n, \Sigma/m_n) \\[8pt] \Sigma | \vec{\mathbf{y}} &\sim \mathcal{W}^{-1} (\nu_n, \mathbf{S}_n) \end{aligned} $$

where

$$ \begin{aligned} \theta_n &= \frac{m \theta_0 n \bar{\mathbf{y}} }{ m_n } \\ m_n &= m + n\\ \nu_n &= \nu_0 + n \\ \mathbf{S}_n &= \mathbf{S}_0 + \mathcal{S} + \frac{mn}{m+n} (\bar{\mathbf{y}} - \theta_0) (\bar{\mathbf{y}} - \theta_0)^T \end{aligned} $$

where $\mathcal{S} := (\vec{\mathbf{y}}-\bar{\mathbf{y}}\mathbb{1})(\vec{\mathbf{y}}-\bar{\mathbf{y}}\mathbb{1})^T$.


PS. Interestingly, we can show that the marginal prior density of the $\theta$ is multivariate $t$ distribution (reference: Bernardo and Smith[1]), where

$$ \theta \sim T(\theta_0, \kappa, q) $$

where $\kappa=\nu_0 \mathbf{S}_0/m, q=2\nu_0 - p + 1$.

Reference

[1] Bernardo and Smith, Bayesian theory(1994), p435

moreblue
  • 1.6k
  • 1
  • 10
  • 26