105

Here is what I usually like doing (for illustration I use the overdispersed and not very easily modelled quine data of pupil's days absent from school from MASS): Test and graph the original count data by plotting observed frequencies and fitted frequencies (see chapter 2 in Friendly) which is supported by the vcd package in R in large parts. For ...


98

Thank you for the interesting question! Difference: One limitation of standard count models is that the zeros and the nonzeros (positives) are assumed to come from the same data-generating process. With hurdle models, these two processes are not constrained to be the same. The basic idea is that a Bernoulli probability governs the binary ...


55

Hurdle models assume that there is only one process by which a zero can be produced, while zero-inflated models assume that there are 2 different processes that can produce a zero. Hurdle models assume 2 types of subjects: (1) those who never experience the outcome and (2) those who always experience the outcome at least once. Zero-inflated models ...


38

Separating the log-likelihood It is correct that most hurdle models can be estimated separately (I would say, instead of sequentially). The reason is that the log-likelihood can be decomposed into two parts that can be maximized separately. This is because $\hat \pi$ is a just a scaling factor in (5.34) that becomes an additive term in the log-likelihood. ...


35

There are a variety of solutions to the case of zero-inflated (semi-)continuous distributions: Tobit regression: assumes that the data come from a single underlying Normal distribution, but that negative values are censored and stacked on zero (e.g. censReg package) hurdle or "two-stage" model: use a binomial model to predict whether the values are 0 or >0, ...


20

You can get the probability of zero-inflation by p <- predict(object, ..., type = "zero") and the mean of the count distribution by lambda <- predict(object, ..., type = "count") See Appendix C of vignette("countreg", package = "pscl") for a few more details. To simulate the distribution, you can either do it manually with ifelse(rbinom(n, size = ...


19

You could use zero- and/or one inflated beta regression models which combine the beta distribution with a degenerate distribution to assign some probability to 0 and 1 respectively. For details see the following references: Ospina, R., & Ferrari, S. L. P. (2010). Inflated beta distributions. Statistical Papers, 51(1), 111-126. Ospina, R., & Ferrari,...


16

Note that the predicted value in a GLM is a mean. For any distribution on non-negative values, to predict a mean of 0, its distribution would have to be entirely a spike at 0. However, with a log-link, you're never going to fit a mean of exactly zero (since that would require $\eta$ to go to $-\infty$). So your problem isn't a problem with the Tweedie, ...


16

Because the loglikelihood contains both $\log(x)$ and $\log(1-x)$, which are unbounded when $x=0$ or $x=1$. See equation (4) of Smithson & Verkuilen, "A Better Lemon Squeezer? Maximum-Likelihood Regression With Beta-Distributed Dependent Variables" (direct link to PDF).


15

I think this is a poorly chosen data set for exploring the advantages of zero inflated models, because, as you note, there isn't that much zero inflation. plot(fitted(fm_pois), fitted(fm_zinb)) shows that the predicted values are almost identical. In data sets with more zero-inflation, the ZI models give different (and usually better fitting) results ...


15

For the approach of using standard diagnostic plots but wanting to know what they should look like, I like the paper: Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne, D.F and Wickham, H. (2009) Statistical Inference for exploratory data analysis and model diagnostics Phil. Trans. R. Soc. A 2009 367, 4361-4383 doi: 10.1098/rsta.2009.0120 ...


15

This is an old question, but I thought it would be useful to add that my DHARMa R package (available from CRAN, see here) now provides standardized residuals for GLMs and GLMMs, based on a simulation approach similar to what is suggested by @GregSnow. From the package description: The DHARMa package uses a simulation-based approach to create readily ...


14

The documentation for the R betareg package mentions that if y also assumes the extremes 0 and 1, a useful transformation in practice is (y * (n−1) + 0.5) / n where n is the sample size. http://cran.r-project.org/web/packages/betareg/vignettes/betareg.pdf They give the reference Smithson M, Verkuilen J (2006). "A Better Lemon Squeezer? Maximum-...


13

Both the Poisson distribution and the geometric distribution are special cases of the negative binomial (NB) distribution. One common notation is that the variance of the NB is $\mu + 1/\theta \cdot \mu^2$ where $\mu$ is the expectation and $\theta$ is responsible for the amount of (over-)dispersion. Sometimes $\alpha = 1/\theta$ is also used. The Poisson ...


13

fit a logistic regression first calculate the probability of zeroes, and then I could remove all the zeroes, and then fit a regular regression using my choice of distribution (poisson e.g.) You're absolutely right. This is one way to fit a zero-inflated model (or as Achim Zeileis points out in the comments, this is strictly a "hurdle model", which one could ...


12

A one-inflated Poisson model for a count $Y_i$ is $$\begin{align}\Pr(Y_i = 1) &= \pi_i +(1-\pi_i)\cdot\mu_i\mathrm{e}^{-\mu_i}\\ \Pr(Y_i = y_i) &= (1-\pi_i)\cdot\frac{\mu_i^{y_i}\mathrm{e}^{-\mu_i}}{y_i!} \qquad \text{when } y_i\neq 1 \end{align}$$ where the Poisson mean $\mu_i$ & Bernoulli probability $\pi_i$ are related to the predictors ...


11

in ZIP model $y_i$~0 with probability $\pi$ and $y_i$~ Poisson($\lambda$) distribution with probability $1-\pi$, thus the ZIP model is mixture model with 2 components and: $$\Pr (y_j = 0) = \pi + (1 - \pi) e^{-\lambda}$$ $$\Pr (y_j = x_i) = (1 - \pi) \frac{\lambda^{x_i} e^{-\lambda}} {x_i!},\qquad x_i \ge 1$$ and in an hurdle model $y_i$~ 0 with ...


10

Predicting the proportion of zeros I am the author of the statmod package and joint author of the tweedie package. Everything in your example is working correctly. The code is accounting correctly for any zeros that might be in the data. As Glen_b and Tim have explained, the predicted mean value will never be exactly zero, unless the probability of a zero ...


9

1) Calculate the mean and the sample variance. $\frac{\bar{X}}{S^2}$ should be $\mathrm{F}(1,n-1)$ distributed, where $n$ is the size of the sample and the process is truly Poisson - since they are independent estimates of the same variance. Note that this test ignores the covariates - so probably not the best way to check over-dispersion in that situation. ...


9

As discussed elsewhere on the site, ordinal regression (e.g., proportional odds, proportional hazards, probit) is a flexible and robust approach. Discontinuities are allowed in the distribution of $Y$, including extreme clumping. Nothing is assumed about the distribution of $Y$ for a single $X$. Zero inflated models make far more assumptions than semi-...


9

A Loess confidence interval doesn't mean much unless the Loess parameters have been cross-validated (which usually is not the case). When you use Loess for exploration, as it was originally intended, understanding how to control it will help you guide your exploration and interpret its results better. Consider this small study of a synthetic dataset which ...


9

The basic idea you describe is a valid approach and it is often called a hurdle model (or two-part model) rather than a zero-inflated model. However, it is crucial that the model for the non-zero data accounts for having the zeros removed. If you fit a Poisson model to the data without zeros this will almost certainly produce a poor fit because the Poisson ...


9

This is certainly possible. The most common definition for a distribution to be right skewed is that the skewness $$ \gamma_1 := E\bigg(\Big(\frac{X-\mu}{\sigma}\Big)^3\bigg) $$ be positive. For instance, the Poisson distribution with parameter $\lambda$ has skewness $\frac{1}{\sqrt{\lambda}}>0$, so it is always right skewed. And for sufficiently small ...


8

Clumping at 0 is called "zero inflation". By far the most common cases are count models, leading to zero-inflated Poisson and zero-inflated negative binomial regression. However, there are ways to model zero inflation with real positive values (e.g. zero-inflated gamma model). See Min and Agresti, 2002, Modelling non negative data with clumping at zero for ...


8

This answer was merged from another thread asking about predictions zero-inflated regression model, but it also applies to the Tweedie GLM model. Regression-like models predict mean of some distribution (normal for linear regression, Bernoulli for logistic regression, Poisson for Poisson regression etc.). In the case of zero-inflated regression you predict ...


8

I am convinced that it is incorrect to use the Vuong test -- in any of its forms -- as a test for zero-inflation. I have had a paper "The misuse of the Vuong test for non-nested models to test for zero-inflation" published that explains why. See http://cybermetrics.wlv.ac.uk/paperdata/misusevuong.pdf. I have also presented the paper at major statistics ...


8

In addition to mgcv and its zero-inflated Poisson families (ziP() and ziplss()), you might also look at the brms package by Paul-Christian Bürkner. It can fit distribution models (where you model more than just the mean, in your case the zero-inflation component of the model can be modelled as a function of covariates just like the count function). You can ...


8

Your training data - just as any other data - is a mixture of signal and noise. In modeling, we try to capture the signal, since the noise is by definition not predictable, except in a probabilistic sense. predict.zeroinfl() by default predicts the expected response, i.e., the signal. Since the noise is mainly variation, stripping this out means that the ...


7

Actually, your second line still works - it just overrides your input $\lambda$ with an estimated $\lambda$. Unless you have good reason to believe your input $\lambda$ is more accurate than the estimate that is done by zeroinfl, I'd just leave it. On the other hand, if you're really firm about using the input $\lambda$, the way to check for a zero ...


7

What ssdecontrol said is very correct. But I'd like to add a few cents to the discussion. I just watched the lecture on Zero Inflated models for count data by Richard McElreath on YouTube. It makes sense to estimate p while controlling for the variables that are explaining the rate of the pure Poisson model, specially if you consider that the chance of an ...


Only top voted, non community-wiki answers of a minimum length are eligible