The data and model used so far:

I have longitudinal count data on 16 pens (8 with treatment A, 8 with treatment B). Each pen houses the same number of animals, and the pens may be considered as independent. On each day (from 1 to 27), a number of diarrhea-related events was counted for each pen. The resulting counts are between 0 and 8, with roughly 45% 0 values and most values below 4.

Some "streaks" (or runs) of zero values (i.e. the counts stay zero for several days in a row) are present in the data. Overall, roughly 60% of the zero values are followed by another zero value on the next day (in both treatment groups).

I did not yet test whether the length and number of streaks is more than expected by chance, but it seems so. (I am not sure how to test this, and I may be wrong with my assessment.)

It is the main aim to characterize the average number of events per day, given the treatment. To do this properly, I also want to account for the temporal autocorrelation (which is present in the data) and to include something like a random intercept for each pen to reflect the possibly different initial health status of the animals in each pen.

I fitted a generalized additive mixed model (GAMM) using the mgcv package in R (Poisson response). I used a smoother (for each treatment group) to model the temporal trend, added a random intercept for each pen and included the residual autocorrelation with an AR(1) approach.

The model is acceptable, shows clear temporal trends which are different for the two groups, modestly different random intercepts per pen and some autocorrelation.

The question: Is there any way to explicitly add to the present model the "streaks" of zero values? I am aware that this is implicitly included in the autocorrelation in the GAMM model above, but it would be really nice to assess whether zero is a special "state". In a sense, I want to know whether the autocorrelation is particularly strong for the zero values.

I thought that maybe one could use a Markov chain approach (if needed, by categorizing the state space e.g. into "0", "1-2", and "3+" so that not too many transition probabilities have to be estimated), but I was not sure how to incorporate the fact that there is a clear temporal trend (and not sure whether and how an inhomogeneous Markov chain approach would make sense).

Any pointer would be greatly appreciated, also to something which would work if I had a larger sample (more pens). I will be happy to add any required clarifications, but I can not share the data at this point.


The mgcv package should be capable of implementing zero-inflated Poisson regression models. This models the distribution as a mixture of a Poisson distribution and a point mass at 0. Implicitly, it is conceptualizing the data as arising from two latent processes, one of which is a Poisson process the other of which only produces 0s, accounting for observing a higher proportion of 0s than one would expect if the data were coming only from a Poisson distribution with a specified mean. An alternative approach would be a hurdle model, which also separates the data into distributions arising from two latent processes, but makes different assumptions about the relationship between those two processes.


A hurdle model assumes that ALL of the observed "0" values are arising from a latent class of respondent that is ONLY capable of producing 0 values, and ALL of the non-zero values are arising from a latent class of respondent that is ONLY capable of producing non-zero values.

On the other hand, a zero-inflated model does not so cleanly separate the two classes. That is, while there is a latent class of respondent that is only capable of producing 0 values, NOT all 0 values observed are necessarily arising from this latent class. The other class of respondent is capable of producing non-zero values AND zeros.

This is an important distinction: in a zero-inflated model you are modeling "excess zero" responses and Poisson-distributed responses which may be zero, while in a hurdle model you are modeling "zero" responses and "non-zero" responses. Which you choose depends on your research questions and the assumptions you are willing to make about your data. For example, in your case you may decide a hurdle model is appropriate since you are interested specifically in the diarrhea and non-diarrhea responses. On the other hand, if you conceptualize diarrhea as simply being one specific observed response of an underlying "sick" state, and it is possible for animals to be sick but not have diarrhea on that specific measurement day, you would want to use a zero-inflated model that allows for this.

Note that this doesn't explicitly model "streaks" or "runs" of 0s as you ask for, but to be honest I don't see why that would be necessary to do so. You could probably come up with some complicated transition/Markov model for this, but I don't know what the utility of that would actually be, or how it would help you answer any questions about the efficacy of the treatment (which I presume is your overall goal). I would say a mixture model like a hurdle or zero-inflated is far more appropriate for the type of data you have.

The down-side to the mixture model approach is that, since these are two-part models, they are more complicated, and force you to make a number of additional assumptions about your data generating process. For example, do you have random effects on BOTH parts of the model (the zero part and the non-zero part)? If so, do they both have the same structure, and are they correlated or uncorrelated? Do you have treatment as a covariate in both parts of the model? etc.

There is a lot of literature on longitudinal zero-inflated models for you to consider. A couple of starting points for you may be:

Alfo, M. and Maruotti, A. "Two-part regression models for longitudinal zero-inflated count data." The Canadian Journal of Statistics, 2010, 38(2). https://onlinelibrary.wiley.com/doi/pdf/10.1002/cjs.10056

Buu, A., Li, R., Tan, X., and Zucker, R.A. "Statistical models for longitudinal zero-inflated count data with applications to the substance abuse field." Statistics in Medicine, 2012, 31(29). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3505239/

  • $\begingroup$ Dear @Ryan Simmons Thank you for your answer, this was very helpful for me. I am aware of zero-inflated models and think it is a good idea to use them to model the data. Also thanks a lot for the clarifications re/ hurdle models and the references. My main motivation for the modeling of streaks was curiosity, and to quantify whether a healthy state is somehow persistent. Thinking about this after your answer again, I am also not sure whether ignoring the streaks if they are present (which is a misspecification of the model) has any relevant deteriorating effect on the parameter estimates. $\endgroup$ – binkyhorse Jul 16 '18 at 14:29
  • $\begingroup$ @binkyhorse the ZiP or hurdle models would only be appropriate if the distribution of the response conditional on the model exhibited more zeroes than expected. It's not clear from the question alone that your data are zero-inflated or have a separate process for the zeroes. The runs of zeroes could be the result of random draws from a Poisson with a small mean. As you are fitting a time-varying mean (the smooth trend), this in principle can account for the "streaks" of zeroes via a low estimated mean count. $\endgroup$ – Gavin Simpson Jul 28 '18 at 12:47
  • $\begingroup$ @GavinSimpson Absolutely, thank you for this remark. I will assess this for the given data set (e.g. by simulation from the model). I still think it would be an interesting general question (maybe not for this small data set) how to properly model data which tend to have streaks. Perhaps one can improve on simple autocorrelation type models. $\endgroup$ – binkyhorse Aug 2 '18 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.