My dataset (with the dependent variable being a count variable) has two features that are different from a standard Poisson regression. I haven't been able to find a more advanced model that accounts for both irregularities, but have found alternatives that can deal with one or the other irregularity. I'll outline the situation below, but my questions are:

  1. Given the features below, is there some hybrid model I could use that simultaneously accounts for both issues?

  2. If not, is my strategy below reasonable in the circumstances?

I have an experiment where people are given a cue word and have to respond with another word that they think other people will pick too. I.e. they have to try coordinate. The dependent count variable is simply the number of people picking each word. The minimum value of this variable, then, is 1. However, while some people were able to coordinate well, many more were not. This means there are a lot of data points at y=1.

A zero-truncated poisson (e.g. in R, vglm(y~x,family=pospoisson)) explicitly models the fact that the minimum count is 1 rather than 0 but doesn't address over-inflation of this minimum count. On the other hand, a hurdle model (e.g. in R hurdle(y~x,dist=poisson)) explicitly models inflated minimum counts, but it expects a minimum count of 0 rather than 1, so to use it for my data, I'd have to use hurdle(y-1~x), so it doesn't address the fact that I have a positive poisson distribution. This doesn't seem conceptually bothersome to me, since a count of 1 (only one person picked the word) means there has been a failure to coordinate.

I don't know of a model that can address both issues simultaneously. I have asked another question here about whether shifting y-1 so that I can use the hurdle model is in principle problematic, or similar enough for this to be informative.

My strategy so far has been to do all three (regular poisson, zero-truncated poisson, hurdle model with shifted values), get boot strapped confidence intervals for the model parameters to show that they're not significantly different, and conclude that my predictor consistently has a significant effect that is similar across these models, and that over-inflation and having a positive poisson distribution thus do not undermine the results of the basic model. Is that a reasonable strategy, or can you recommend something better?

The dependent count variable ranges from 1 to 18 (20 people were given each cue word, but there were no cases where all 20 gave the same response), with most of the weight at 1. There is one predictor, which ranges continuously from 0 to 1. For extremely low values of that predictor, I expect no coordination (y=1 meaning only one person generated the response). For low values of the predictor, I expect small amounts of coordination, and more people should coordinate over a response as the value of the predictor increases.

  • $\begingroup$ It would be helpful if you define the content of records used for Poisson regression runs: what is the range of the dependent, what are the independent features? You never stated whether counts for the dependent rate parameter can take on values of 5, 100, 10^6, etc. What are your independent features, their scales, ranges, etc? You may not need to perform Poisson regression, because it's only used for rare outcomes, when $npq \leq 5$. You need to also define a success or failure, and not pad records with a one simply because a group spawned a word. Define $y$ on # of successes. $\endgroup$ – JoleT Aug 12 '16 at 16:10
  • $\begingroup$ @PEL - I've added some info about the dependent and predictor variables. Could you please unpack for me what npq≤5 means. The fact that just one person picked a certain response is of interest (out of 10000s English words, very few were given as responses, so having a count of 1 is not the same as nobody picking a response), so I don't think this counts as padding. $\endgroup$ – Justin Aug 12 '16 at 16:27
  • $\begingroup$ You might normalize the dependent (or get the percentiles from ranks) and then plot and correlate $y$ with $x$. You can also recode $x$ into a new binary(0,1), transform to 1,2 then look at the mean of $y$ within each group. The dependent variable in Poisson regression is always (internally) the success(failure) rate parameter $\lambda_i=c/t$, where $c$ is the number of successes, and $t$ is the time over observation, area, or other integral. This is commonly expressed he same way for the both the binomial and Poisson mean, $\mu=np \rightarrow c=\lambda t$. $\endgroup$ – JoleT Aug 12 '16 at 16:39
  • $\begingroup$ Thus, sum all of the $y$'s for all groups to get $\mu$, then divide by the number of groups $n$, which gives you $p$. Next, multiply $np(1-p)$ and check to see if it's less than 5. If not, your mean $\mu$ is binomially distributed $B(n,p)$. The Poisson distribution has only one parameter, the mean, $P(\mu)$, which is equal to the variance. If your data are binomial, then use a binomial-based regression. $\endgroup$ – JoleT Aug 12 '16 at 16:48
  • $\begingroup$ I see no problem with shifting y by 1. Why do you think that might be problematic? $\endgroup$ – Peter Flom Aug 13 '16 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.