The posterior distribution for weights in linear regression setup is

\begin{equation} \begin{aligned} B &| Y,X \sim \mathcal{N}(\mu, \Lambda) \\ \mu &= \Lambda X^{\mathsf{T}}\Sigma^{-1}Y \\ \Lambda &= (X^{\mathsf{T}}\Sigma^{-1}X + \Omega^{-1})^{-1} \end{aligned} \end{equation}

$\Sigma$ is the covariance matrix, $\sigma_{\epsilon}\mathbf{I}$, where $\sigma_{\epsilon}$ is noice on each measurement $y = x \beta + \epsilon$, and $\Omega$ is the covariance on the prior on $\beta$, with $\sigma_{\beta}$ on its diagonal. $B \sim \mathcal{N}(0, \sigma_{\beta})$.

My question is, how to estimate $\Omega$ and $\Sigma$ when both are unknown. Should I assume an iterative process, where every time I replace $\mu$ and $\Lambda$ with $\Sigma$ and $\Omega$ from the previous steps ?

  • $\begingroup$ If $\Sigma$ is unknown it should be associated with a prior. Same thing for $\Omega$. $\endgroup$ – Xi'an Feb 6 '15 at 19:16
  • $\begingroup$ I know usually there is a inverse gamma prior. But, in Bishop book (page 153) I do not see such thing. $\endgroup$ – user4581 Feb 6 '15 at 19:18
  • $\begingroup$ There is no "default" priors and Gelman argues that the common Gamma(0.001, 0.001) is not the best choice... $\endgroup$ – Tim Feb 6 '15 at 21:17
  • $\begingroup$ Thanks for sharing the paper; I am following Bishop book. I don't know if I can provide a link to the book. $\endgroup$ – user4581 Feb 7 '15 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.