Downsides of inverse Wishart prior in hierarchical models

Question

I am working with a Bayesian hierarchical model that has a number of parameters for each experimental unit (6 parameters). I really do not know all that much about them a-priori, but it is quite plausible that they could somehow be correlated.

Thus, I was considering a multivariate normal distribution for the random effects with a pretty uninformative prior on the mean of the random effects and some prior for the covariance matrix. An inverse Wishart with, say, dimension + 1 degrees of freedom (in my example = 7) and the identity matrix as the scale matrix may seem logical, because it is available by default in my software packages and is supposedly relatively uninformative.

I have seen in Gelman's Bayesian Data Analysis book that another option could be to use a separate random effects standard deviation for each parameter in combination with an inverse Wishart with the idea that one could specify more informative priors on the random effects standard deviations (while keeping the correlation structure more uninformative; in addition there's the LKJ prior, which as far as I know is only implemented in Stan.). What I am wondering is in part whether I would really need to do that when I do not know much a-priori and expect to have enough data that hopefully the data would dominate whatever prior I specify (as long as the prior distribution allows sufficient flexibility that this actually happens).

Answer 1

Here are some relevant resources (full disclosure: the first link is to a paper of mine):

• http://newprairiepress.org/agstatconference/2014/proceedings/8
• http://www.themattsimpson.com/2012/08/20/prior-distributions-for-covariance-matrices-the-scaled-inverse-wishart-prior/
• http://andrewgelman.com/2012/08/29/more-on-scaled-inverse-wishart-and-prior-independence/

A prior over a covariance matrix can be considered as a joint prior over the variances, i.e. the diagonals of the covariance matrix, and the correlations, i.e. the off-diagonal elements divided by the square root of the row and column diagonal elements.

In my opinion, the problems with an IW prior are

1. the uncertainty for all variance parameters are controlled by the single degree of freedom parameter,
2. the marginal distribution for the variance is an inverse gamma (the IW(7,I) implies a marginal IG(1,1/2)) which has a region near zero with extremely low density which causes a bias toward larger variances when the true variance is small, and
3. there is a prior dependency between the variances and correlations such that larger variances are associated with correlations near +/- 1 while small variances are associated with correlations near zero. Thus, when the true variance is small, the correlation will be estimated to be zero regardless of the true value of the correlation and this bias remains even for relatively large sample sizes.

From your description, problem #1 is not so relevant, but problems 2 and 3 could be relevant. Although using more sophisticated priors will resolve this issue, a pragmatic solution is to think more carefully about the scale matrix in the IW prior. Instead of using the identity matrix, use a diagonal matrix with values for each element that a reasonable given the data you are analyzing. Alternatively, you could perform a prior sensitivity analysis by trying scale matrices of the form $\epsilon$ times the identity matrix.

The above discussion primarily focused on the issues of using an inverse Wishart distribution for any covariance matrix. There is an additional concern when using the inverse Wishart as the prior for a hierarchical covariance matrix. In Gelman (2006), the inverse gamma is shown to be informative for a hierarchical variance and this issue will carry over to an inverse Wishart on a hierarchical covariance matrix. The suggestion in that paper is to use half-Cauchy distributions (or uniforms) on the hierarchical standard deviations. If you separately define priors for the hierarchical standard deviations and correlations, then you will still need a prior over a correlation matrix, e.g. the LKJ prior.

So yes, I think you really need to think carefully about this prior and perform a sensitivity analysis to determine how impactful the prior is. With enough data, the likelihood should be able to overwhelm the prior, but it is unclear how much data is enough.