I know some bayesian methods employ an inverse wishart distribution for the prior distribution of the covariance matrix in a linear regression. I.e. for the model:


Where $\epsilon \sim N(0, \Sigma)$, some methods will use $\pi(\Sigma)\sim W^{-1}(A,v)$. My question is however, what should the matrices/values $A$ and $v $ be? I read somewhere that $v$ was degrees of freedom but could not work out of what... should $v$ be $n-p$ where $X$ is $(n \times p)$ and how should $A$ be chosen?


1 Answer 1


A textbook sort of answer would be that you as the data analyst should choose $A$ and $\nu$ to best capture your prior beliefs about the unobserved covariance $\Sigma$. Here's one approach to doing that:

You first recognize that if an $n\times n$ matrix $\Sigma$ has the Inverse Wishart distribution $W^{-1}(A,\,\nu)$, then its mean is $$E[\Sigma]=\frac{1}{\nu-n-1}A\quad \text{if }\nu>n+1.$$ So for starters, you might want to pick $\nu>n+1$ so that the mean of your prior exists. Then, you could decide what you believe about the "average behavior" of $\Sigma$ in your application and then select $$A=(\nu-n-1)\bar{\Sigma},$$ where $\bar{\Sigma}$ is what you think the mean is.

Furthermore, you can look up expressions for the variances and covariances of the elements of $\Sigma$. In all cases, you will notice that as the degrees of freedom $\nu$ increase, the variances shrink. So if you believe that the prior distribution for $\Sigma$ should be diffuse, you might want to keep $\nu$ low. If you think it should be tight, you'll want to pick a bigger $\nu$.

Update: picking $\bar{\Sigma}$

One approach is to set the prior mean of $\Sigma$ to some "reasonable" estimate taken from past research. Perhaps there are important papers in the literature related to your application which will give you an idea of where it makes sense to center $\Sigma$.

If you don't have such knowledge, but you have a moderate to large dataset, another approach is to take your dataset $(y_i,\,x_i)_{i=1}^N$ and break it into two subsets: a "training" set $(y_j,\,x_j)_{j=1}^{N_{tr}}$ and a "test" set $(y_k,\,x_k)_{k=1}^{N_{te}}$, where $N_{tr}$ is small compared to $N_{te}$ and $N_{tr}+N_{te}=N$. You can use the training dataset to compute, for example, the MLE estimate of $\Sigma$, use that as the prior mean of $\Sigma$, and then conduct your data analysis using the remaining test data. That is, you are setting aside a portion of your data in order to "train" the prior, and you are then ignoring it when you do the main estimation. You wouldn't train the prior on a portion of the data but then also include that data in the main estimation.

  • $\begingroup$ I see thank you, that is very helpful! $\endgroup$
    – JDoe2
    Commented Aug 1, 2019 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.