I have a hierarchical model that I need to validate. My model is as follows: we have a collection of $\lambda_i$ that we draw from $Gamma(\alpha,\beta)$. Then, we draw our data point $y_i$ from $Poisson(\lambda_i)$. I get a distribution of $\alpha,\beta,\{\lambda_i\}$ via a Gibbs sampler combined with a Metropolis step. This part is fine. My question is how do I validate such a model? I have my set of data, each one corresponding to one particular $\lambda_i$. I'm not sure what statistical tests/ other steps I should take to check are.


migrated from math.stackexchange.com Apr 23 '12 at 16:03

This question came from our site for people studying math at any level and professionals in related fields.

  • $\begingroup$ I am not aware of a general Bayesian model validation technique. It is more common to conduct model comparison via Bayes factor, Scoring rules such as the log-predictive scores, and etcetera. $\endgroup$ – user10525 Apr 23 '12 at 7:30
  • $\begingroup$ Perhaps, chapter 24 of Gelman and Hill on Model checking and comparison might be useful. They talk a bit about posterior predictive checking. $\endgroup$ – Jeromy Anglim Apr 23 '12 at 7:36
  • $\begingroup$ I was thinking of generating some estimated values for the $y_i$ and then using some Chi-square tests, do you think that would be good? $\endgroup$ – Andrew Apr 23 '12 at 7:36
  • $\begingroup$ Chi-square test: "A test of goodness of fit establishes whether or not an observed frequency distribution differs from a theoretical distribution". If you simulate from model $M$ and then try to check if these observations come from model $M$, guess what is going to happen ... $\endgroup$ – user10525 Apr 23 '12 at 7:52
  • $\begingroup$ I think I may have been unclear. I have actual data observations. I also have a joint distribution I got from Metropolis/Gibbs that I can sample from to generate values. I would then compare these generated values to the observed data. $\endgroup$ – Andrew Apr 23 '12 at 7:59

Posterior predictive checks, outlined in Gelman et al (1996), are an obvious starting point. Given how simple the model is, it probably makes sense to use graphical checks. Plot the histogram of $(y_1,...,y_n)$ against histograms of several posterior predictive replications $(y_1^{rep},...,y_n^{rep})$. If you spot a feature that doesn't fit, you can formalize things by defining an appropriate discrepancy statistic and computing the posterior predictive p-value of your model against that statistic.


Perhaps posterior predictive checking (i.e., comparing your data to predictions generated by the model). For instance along these lines:


  • $\begingroup$ Sorry nobody saw this sooner. I like the reference. Welcome to the site. $\endgroup$ – Placidia Oct 20 '12 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.