How to test for Zero-Inflation in a dataset?

Question

I have a dataset which seems to have a lot of zeroes. I have already fit a poisson regression model as well as a negative binomial model. I would like to fit zero-inflated and hurdle models as well.

Before I do I would like to run a test to investigate whether my data really is zero inflated. What test(s) is/are there to determine whether my data are zero-inflated?

Answer 1

The score test (referenced in the comments by Ben Bolker) is performed by first calculating the rate estimate $\hat{\lambda}= \bar{x}$. Then count the number of observed 0s denoted $n_0$ and the total number of observations $n$. Calculate $\tilde{p}_0=\exp[-\hat{\lambda}]$. Then the test statistic is calculated by the formula: $\frac{(n_0 - n\tilde{p}_0 )^2}{n\tilde{p}_0(1-\tilde{p}_0) - n\bar{x}\tilde{p}_0^2}$. This test statistic has a $\chi^2_1$ distribution which can be looked up in tables or via statistical software.

Here is some R code that will do this:

pois_data <-rpois(100,lambda=1)
lambda_est <- mean(pois_data)

p0_tilde <- exp(-lambda_est)
p0_tilde
n0 <- sum(1*(!(pois_data >0)))
n <- length(pois_data)

# number of observtions 'expected' to be zero
n*p0_tilde

#now lets perform the JVDB score test 
numerator <- (n0 -n*p0_tilde)^2
denominator <- n*p0_tilde*(1-p0_tilde) - n*lambda_est*(p0_tilde^2)

test_stat <- numerator/denominator

pvalue <- pchisq(test_stat,df=1, ncp=0, lower.tail=FALSE)
pvalue 

Answer 2

I think there are different ways to do this. One thing you can do is to compare a zero-inflated negative binomial/Poisson model with its regular binomial/Poisson counter part without the zero-inflation component. It would look like this in R:

zinb <- read.csv("http://www.ats.ucla.edu/stat/data/fish.csv")
zinb <- within(zinb, {
  nofish <- factor(nofish)
  livebait <- factor(livebait)
  camper <- factor(camper)
})

require(pscl)
require(MASS)
require(boot)

## fit a negative binomial model
m1 <- glm.nb(count ~ child + camper, data = zinb)

## fit a zero-inflated negative binomial model
m1_zi <- zeroinfl(count ~ child + camper| persons,
               data = zinb, dist = "negbin", EM = TRUE)
## compare 2 models
vuong(m1, m1_zi)

For more information, see this ever useful tutorial.

Answer 3

Consider some model $f(x)$. If we want to turn $f(x)$ into a zero-inflated model, then we define $g(x)$ to equal $f(x)$ with proportion $p$ and to equal $0$ with proportion $1-p$.

In this case, there are two processes at work here. One process generates only zeros and one process generates results from $f(x)$. My understanding is that a zero-inflated model is only appropriate when there is an alternate process that generates only zeros. For example, if you are attempting to estimate the number of widgets different stores sell, but some stores do not have widgets for sale, then it seems like two processes are at work here: one process that generates only zeros (those stores that cannot sell widgets because they do not ever stock widgets for sale) and another process that generates different values (those stores that do stock widgets and therefore can sell some).

Rather than having a "test" to determine whether the data are zero-inflated, I would suggest determining whether it is plausible that there are two processes at work - one being a zero-generating process at work and another process that generates non-zero numbers. If it seems reasonable given the context of your data, then use a zero-inflated model. If it doesn't seem reasonable given the context of your data, then a zero-inflated model is probably inappropriate even though it may appear to fit your data better.

(It might not be clear from what I've written above, but I want to articulate the fact that both processes can generate zeros. One process generates only zeros and the other process can generate different values which may be zero. For example, a store can stock widgets and happen to sell zero widgets. This is different from a store that does not stock widgets and therefore must sell zero widgets by default.)