# Is there a desription in the literature of a Normal hierarchical model with hyperparameters for both the mean and the standard deviation?

I'm looking for a comprehensive description of and justification for a Normal hierarchical model where both the means of the groups and the standard deviation are modelled. It is common to find something like the following model in many textbooks (e.g. Gelman et al., p. 288):

$$y_{ij} \sim \text{Normal}(\mu_i,\sigma) \\ \mu_i \sim \text{Normal}(M, S)$$

where $y_{ij}$ is the $i$th datapoint from group $j$ and where non-informative priors are proposed for $M,S$ and $\sigma$. What I'm looking for is an extension of this model where also the standard deviations of each group are modelled and given hyperparameters (and not only a single $\sigma$ is assumed for all groups). That is something like:

$$y_{ij} \sim \text{Normal}(\mu_i,\sigma_i) \\ \mu_i \sim \text{Normal}(M, S) \\ \sigma_i \sim \text{SomeDistribution}(P_1,P_2,\dots)\\$$

but where proposals are given for

• The distribution of the $\sigma_i$s ($\text{SomeDistribution}$ in the model above).
• Non-informative prior distributions for the hyperparameters $M, S$ and the parameters of $\text{SomeDistribution}$.

I have not been able to find this in the literature, and my question is: Where can I find this model described in the literature? Or alternatively: What should such a model look like?

## References

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis. CRC press.

• Is this helpful? More discussion here. – conjugateprior Aug 18 '14 at 9:51
• @conjugateprior In general, yes, but for my particular question, no. It still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. – Rasmus Bååth Aug 18 '14 at 10:52
• Ah, I see what you mean. Now I'm wondering whether this model be easily identified if you formulated it. But I've no intuition either way tbh. – conjugateprior Aug 18 '14 at 11:04
• I think that ch. 7 of the Jackman textbook has a hierarchical ANOVA model that has heterogenous bottom level variances, but I don't have it to hand. – conjugateprior Aug 18 '14 at 11:29
• My first instinct would be to use the same prior family for per-group variance as for pooled variance. – shadowtalker Aug 21 '14 at 15:32

The model you describe is discussed by Gelman in section 6 here, where the group variances $\sigma_k$ have a half-Cauchy prior multiplied by a scale factor $A$.

I'm only seeing this independent variance model in the context of Hierarchical regression models.

If you look at chapter 5 of Gelman's Bayesian Data Analysis he alludes to the possibility of including independent variance components in a hierarchical model (reflected in his notation), but sets it aside and says it will be considered later.

He finally discusses it in the context of Hierarchical Linear models in chapter 15. He notes that the same hiearchical model studied in ch. 5 can be represented as a Hierarchical linear regression problem. The model in (15.2) tries to predict the proportion of Democratic voters in a presidential election given some past data at the state level. He adds an additional level to the model to capture regional patterns in the voting data which partitions states (groups) into regional clusters--southern and non-southern. The within region variance is modeled as an independent draw from a uniform distribution.

The incorporation of independent variance parameters is at a higher level than the model you described, but it might help direct your search for more applicable examples.

• As far as I can see it still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. Or perhaps I'm missing something? – Rasmus Bååth Aug 25 '14 at 21:27
• @RasmusBååth, I'm pretty sure the model in section 6 of that paper is an instance of what you want. Also, it's discussed in Ch. 15 of Gelman's book. See my edits above for more info. – jerad Aug 28 '14 at 11:48
• Thank you for pointing this out! I don't' know how I missed it... Using the half-Cauchy as a hyper distribution is a little bit strange in that it can "pull down" the estimates of large standard deviations but not "pull up" small standard deviations. It would feel more natural for me to use a (potentially) heap shaped distribution like a gamma or log-normal... – Rasmus Bååth Aug 28 '14 at 15:25

Maybe this is not an answer, but it is too long to go on the comment.

I do not have reference, but in your case i will suggest $\frac{1}{\sigma_i^2} \sim G (a,b)$ (note that i use the precision and not the standard deviation) where G is the gamma distribution. IN this way you can easily compute the full conditional for all the $\sigma_i^2$. For the hyperparameters i would use $M \sim N()$ and $S \sim IG()$, again with this choice you have closed form for the full conditional, if you want them non informative you can use distribution with high variance. For the parameters $(a,b)$ you can use the distribution suggested in http://en.wikipedia.org/wiki/Conjugate_prior at the gamma section.

All the full conditional distribution can be derived in closed form and maybe you can compute exactly the posterior without MCMC algorithm

• Agreed. You can also show that a hierarchical prior on sigma is inverse-gamma distributed, but that might be harder to work with. Also, check "Doing Bayesian Data Analysis" by Kruschke. Great book. Might have the example you're looking for. – Nate Aug 19 '14 at 0:36
• In Kruschkes "Doing Bayesian Data Analysis" there is indeed a "fully hierarchical" model described on page 330 where he uses a Gamma to model the precisions (inverse variances) of the groups. He does not give any concrete advice regarding the hyperpriors other than that they "must be appropriate for the domain and data at hand". – Rasmus Bååth Aug 25 '14 at 21:51

The model you mention has been largely studied. One of the most popular references is:

http://www.jstor.org/stable/2291572

where the model (Equation 5) is studied with improper priors that lead to closed-form conditionals, allowing for the construction of a Gibbs sampler. Please, take a look at the paper for further details and additional references.

• As far as I can see it still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. Or perhaps I'm missing something? – Rasmus Bååth Aug 22 '14 at 10:33