Strategy for deciding appropriate model for count data

What is the appropriate strategy for deciding which model to use with count data? I have count data that i need to model as a multilevel model and it was recommended to me (on this site) that the best way to do so this is through bugs or MCMCglmm. However i am still trying to learn about bayesian statistics, and i thought i should first try to fit my data as generalized linear models and ignore the nested structure of the data (just so i can get a vague idea of what to expect).

About 70% of the data are 0 and the ratio of variance to the mean is 33. So the data is quite over-dispersed.

After trying a number of different options (including poisson, negative binomial, quassi and zero inflated model) i see very little consistency in the results (varying from everything is significant to nothing is significant).

How can i go about making an informed decision about which type of model to choose based on the 0 inflation and over-dispersion? For instance, how can i infer that quassi-poisson is more appropriate than negative binomial (or vise versa) and how can i know that using either has dealt adequately (or not) with the excess zeros? Similarly, how do i evaluate that there is no more over-dispersion if a zero-inflated model is used? or how should i decide between a zero inflated poisson and a zero inflated negative binomial?

-

You can always compare count models by looking at their predictions (preferrably on a hold out set). J. Scott Long discusses this graphically (plotting the predicted values against actuals). His text book here describes in details but you can also look at 6.4 on this document.

You can compare models using AIC or BIC and there is also a test called Voung test that I am not terribly familiar with but can compare zero inflated to non nested models. Here is a Sas paper describing it briefly on page 10 to get you started. It also is implmented in R posting

-
Thanks for the advice. I will definitely try to examine the predictions before deciding on the model –  George Michaelides Feb 25 '11 at 14:41

A couple things to add to what B_Miner said:

1) You wrote that the models varied from "everything significant" to "nothing significant" but this is not a good way to compare models. Look, instead, at predicted values (as B_miner suggested) and effect sizes.

2) If 70% of the data are 0, I can't imagine that a model without 0 inflation is appropriate.

3) Even if you don't want to go Bayesian, you can use GLMMs in SAS (PROC GLIMMIX or NLMIXED) and in R (various packages). Ignoring the nested nature may mess everything up.

4) In general, deciding on which model is best is an art, not a science. There are statistics to use, but they are a guide to judgment. Just looking at what you wrote, I would say a ZINB model looks right

-
The intention is that i will eventually try to model this using Bayesian, but i was trying to understand how i can make a decision before fitting the models. If there is a possibility that ignoring the nested nature of the data messes things up, them i will try them GLMMs first. The only package for R that i am aware of that can do multilevel ZINB is glmmADMB. Would you recommend any other packages? –  George Michaelides Feb 25 '11 at 14:48
if you want to fit a ZINB with R, get the package pscl and try to use the command zeroinfl(<model>, dist=negative). For further information see ?zeroinfl after loading the required package!