Strategy for deciding appropriate model for count data

Question

What is the appropriate strategy for deciding which model to use with count data? I have count data that i need to model as a multilevel model and it was recommended to me (on this site) that the best way to do so this is through bugs or MCMCglmm. However i am still trying to learn about bayesian statistics, and i thought i should first try to fit my data as generalized linear models and ignore the nested structure of the data (just so i can get a vague idea of what to expect).

About 70% of the data are 0 and the ratio of variance to the mean is 33. So the data is quite over-dispersed.

After trying a number of different options (including poisson, negative binomial, quassi and zero inflated model) i see very little consistency in the results (varying from everything is significant to nothing is significant).

How can i go about making an informed decision about which type of model to choose based on the 0 inflation and over-dispersion? For instance, how can i infer that quassi-poisson is more appropriate than negative binomial (or vise versa) and how can i know that using either has dealt adequately (or not) with the excess zeros? Similarly, how do i evaluate that there is no more over-dispersion if a zero-inflated model is used? or how should i decide between a zero inflated poisson and a zero inflated negative binomial?

Answer 1

You can always compare count models by looking at their predictions (preferrably on a hold out set). J. Scott Long discusses this graphically (plotting the predicted values against actuals). His text book here describes in details but you can also look at 6.4 on this document.

You can compare models using AIC or BIC and there is also a test called Voung test that I am not terribly familiar with but can compare zero inflated to non nested models. Here is a Sas paper describing it briefly on page 10 to get you started. It also is implmented in R posting

Answer 2

A couple things to add to what B_Miner said:

1) You wrote that the models varied from "everything significant" to "nothing significant" but this is not a good way to compare models. Look, instead, at predicted values (as B_miner suggested) and effect sizes.

2) If 70% of the data are 0, I can't imagine that a model without 0 inflation is appropriate.

3) Even if you don't want to go Bayesian, you can use GLMMs in SAS (PROC GLIMMIX or NLMIXED) and in R (various packages). Ignoring the nested nature may mess everything up.

4) In general, deciding on which model is best is an art, not a science. There are statistics to use, but they are a guide to judgment. Just looking at what you wrote, I would say a ZINB model looks right

Answer 3

My understanding is that zero-inflated distributions should be used when there is a rationale for certain items to produce counts of zeroes versus any other count. In other words, a zero-inflated distribution should be used if the zeroes are produced by a separate process than the one producing the other counts. If you have no rationale for this, given the overdispersion in your sample, I suggest using a negative binomial distribution because it accurately represents the abundance of zeroes and it represents unobserved heterogeneity by freely estimating this parameter. As mentioned above, Scott Long's book is a great reference.

Answer 4

absolutely agreed to what Matt said, first you have to think about the background of the data... It doesn't make any sense to fit ZI models, when there are no Zero generating triggers in the population! The advantage of NB models are that they can display unobserved heterogenity in a gamma distributed random variable. Techniqually: The main reasons for overdispersion are unobs Heterogenity and Zero inflation. I do not believe that your fit is bad. Btw to get the goodness of fit you should always compare the Deviance with the degrees of freedom of your model. If the Deviance D is higher than n-(p+1) (this is df) than you should search a better model. Although there are mostly no better models than ZINB to get rid of overdispersion.

if you want to fit a ZINB with R, get the package pscl and try to use the command zeroinfl(<model>, dist=negative). For further information see ?zeroinfl after loading the required package!