4
$\begingroup$

I'm looking for a comprehensive description of and justification for a Normal hierarchical model where both the means of the groups and the standard deviation are modelled. It is common to find something like the following model in many textbooks (e.g. Gelman et al., p. 288):

$$y_{ij} \sim \text{Normal}(\mu_i,\sigma) \\ \mu_i \sim \text{Normal}(M, S)$$

where $y_{ij}$ is the $i$th datapoint from group $j$ and where non-informative priors are proposed for $M,S$ and $\sigma$. What I'm looking for is an extension of this model where also the standard deviations of each group are modelled and given hyperparameters (and not only a single $\sigma$ is assumed for all groups). That is something like:

$$y_{ij} \sim \text{Normal}(\mu_i,\sigma_i) \\ \mu_i \sim \text{Normal}(M, S) \\ \sigma_i \sim \text{SomeDistribution}(P_1,P_2,\dots)\\ $$

but where proposals are given for

  • The distribution of the $\sigma_i$s ($\text{SomeDistribution}$ in the model above).
  • Non-informative prior distributions for the hyperparameters $M, S$ and the parameters of $\text{SomeDistribution}$.

I have not been able to find this in the literature, and my question is: Where can I find this model described in the literature? Or alternatively: What should such a model look like?

References

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis. CRC press.

$\endgroup$
  • $\begingroup$ Is this helpful? More discussion here. $\endgroup$ – conjugateprior Aug 18 '14 at 9:51
  • $\begingroup$ @conjugateprior In general, yes, but for my particular question, no. It still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. $\endgroup$ – Rasmus Bååth Aug 18 '14 at 10:52
  • $\begingroup$ Ah, I see what you mean. Now I'm wondering whether this model be easily identified if you formulated it. But I've no intuition either way tbh. $\endgroup$ – conjugateprior Aug 18 '14 at 11:04
  • $\begingroup$ I think that ch. 7 of the Jackman textbook has a hierarchical ANOVA model that has heterogenous bottom level variances, but I don't have it to hand. $\endgroup$ – conjugateprior Aug 18 '14 at 11:29
  • $\begingroup$ My first instinct would be to use the same prior family for per-group variance as for pooled variance. $\endgroup$ – shadowtalker Aug 21 '14 at 15:32
1
+50
$\begingroup$

The model you describe is discussed by Gelman in section 6 here, where the group variances $\sigma_k$ have a half-Cauchy prior multiplied by a scale factor $A$.

I'm only seeing this independent variance model in the context of Hierarchical regression models.

If you look at chapter 5 of Gelman's Bayesian Data Analysis he alludes to the possibility of including independent variance components in a hierarchical model (reflected in his notation), but sets it aside and says it will be considered later.

He finally discusses it in the context of Hierarchical Linear models in chapter 15. He notes that the same hiearchical model studied in ch. 5 can be represented as a Hierarchical linear regression problem. The model in (15.2) tries to predict the proportion of Democratic voters in a presidential election given some past data at the state level. He adds an additional level to the model to capture regional patterns in the voting data which partitions states (groups) into regional clusters--southern and non-southern. The within region variance is modeled as an independent draw from a uniform distribution.

enter image description here

The incorporation of independent variance parameters is at a higher level than the model you described, but it might help direct your search for more applicable examples.

$\endgroup$
  • $\begingroup$ As far as I can see it still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. Or perhaps I'm missing something? $\endgroup$ – Rasmus Bååth Aug 25 '14 at 21:27
  • $\begingroup$ @RasmusBååth, I'm pretty sure the model in section 6 of that paper is an instance of what you want. Also, it's discussed in Ch. 15 of Gelman's book. See my edits above for more info. $\endgroup$ – jerad Aug 28 '14 at 11:48
  • $\begingroup$ Thank you for pointing this out! I don't' know how I missed it... Using the half-Cauchy as a hyper distribution is a little bit strange in that it can "pull down" the estimates of large standard deviations but not "pull up" small standard deviations. It would feel more natural for me to use a (potentially) heap shaped distribution like a gamma or log-normal... $\endgroup$ – Rasmus Bååth Aug 28 '14 at 15:25
0
$\begingroup$

Maybe this is not an answer, but it is too long to go on the comment.

I do not have reference, but in your case i will suggest $\frac{1}{\sigma_i^2} \sim G (a,b)$ (note that i use the precision and not the standard deviation) where G is the gamma distribution. IN this way you can easily compute the full conditional for all the $\sigma_i^2$. For the hyperparameters i would use $M \sim N()$ and $S \sim IG()$, again with this choice you have closed form for the full conditional, if you want them non informative you can use distribution with high variance. For the parameters $(a,b)$ you can use the distribution suggested in http://en.wikipedia.org/wiki/Conjugate_prior at the gamma section.

All the full conditional distribution can be derived in closed form and maybe you can compute exactly the posterior without MCMC algorithm

$\endgroup$
  • 2
    $\begingroup$ Agreed. You can also show that a hierarchical prior on sigma is inverse-gamma distributed, but that might be harder to work with. Also, check "Doing Bayesian Data Analysis" by Kruschke. Great book. Might have the example you're looking for. $\endgroup$ – Nate Aug 19 '14 at 0:36
  • $\begingroup$ In Kruschkes "Doing Bayesian Data Analysis" there is indeed a "fully hierarchical" model described on page 330 where he uses a Gamma to model the precisions (inverse variances) of the groups. He does not give any concrete advice regarding the hyperpriors other than that they "must be appropriate for the domain and data at hand". $\endgroup$ – Rasmus Bååth Aug 25 '14 at 21:51
0
$\begingroup$

The model you mention has been largely studied. One of the most popular references is:

http://www.jstor.org/stable/2291572

(open access) http://www.ime.unicamp.br/~lramos/mi667/ref/15casella96.pdf

where the model (Equation 5) is studied with improper priors that lead to closed-form conditionals, allowing for the construction of a Gibbs sampler. Please, take a look at the paper for further details and additional references.

$\endgroup$
  • $\begingroup$ As far as I can see it still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. Or perhaps I'm missing something? $\endgroup$ – Rasmus Bååth Aug 22 '14 at 10:33
0
$\begingroup$

I hope I can help you!

I suggest you to look the following references to have more information about hierarchical models and hyperparameters:

Edward Greenberg - Introduction to Bayesian Econometrics, 4.6 Hierarchical Models

Gary Koop - Bayesian Econometrics (several explainations from page 140)

Giannone, Lenza, Primiceri - Prior Selection for Vector Autoregressions

$\endgroup$
  • $\begingroup$ I have checked the Greenberg and the Giannone et al. reference but found no mention of a model where the standard deviation of the groups are modelled, like in the last model sketch in my question. The Koop book I don't have access to at the moment. $\endgroup$ – Rasmus Bååth Aug 25 '14 at 21:32
  • $\begingroup$ I'm sorry, I think I misunderstood you. I suggest you to read the Koop book and Andrew Gelman, Prior distributions for variance parameters in hierarchical models $\endgroup$ – user40899 Aug 25 '14 at 23:46
  • $\begingroup$ The Gelman paper has been suggested by others answering this question but unfortunately it is about the variance parameter of the prior distribution of the means, it does not discuss how to model the standard deviations of the groups. $\endgroup$ – Rasmus Bååth Aug 26 '14 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.