4
$\begingroup$

Is there anything wrong with modeling zero-containing non-negative data by first multiplying it by 10^something, then rounding it, and then modeling it as poisson (or negative binomial)?

In my case, I've got data on crop yields that includes complete crop failures, and response vs fitted plots (from OLS) show a lot of fitted values below zero. Furthermore, the measurements are noisy, so I won't really be losing much information by rounding. So multiplying, rounding, and modeling as a count variable seems attractive.

This is longitudinal data with fixed intercepts (to control for all stable covariates), so tobit is out because it also isn't robust to the incidental parameters problem. Adding a small constant and then logging gives ugly residual plots. Gamma is out because gamma regression also isn't robust to the incidental parameters problem, and I've got a lot of fixed intercepts.

I've come across this paper by Paul Allison which shows that poisson and negative binomial with fixed intercepts gives roughly consistent estimates of $\beta$, but that it has too-small SE's.

So, is there something wrong with multiplying data by 100, rounding and treating it as count? Or are there better approaches?

Edit: One idea that has come to me, but that nobody has recommended, is to use a Tweedie distribution, with software that estimates $p$. But again it's not clear that this is robust to the incidental parameters problem.

$\endgroup$
7
  • $\begingroup$ Gamma can't handle zeros. And in my case, multiply by 100 and then rounding is a fine approximation of the true value, which is measured with noise. $\endgroup$ Mar 15, 2014 at 0:48
  • $\begingroup$ Also, FE poisson gives consistent coefficients (though wrong SE's), while I don't know if FE gamma suffers from the incidental parameter problem. $\endgroup$ Mar 15, 2014 at 0:49
  • 2
    $\begingroup$ How many zero observations are there? One solution may be a two part regression, where you first model yield=0 or yield>0 with a logistic regression, then model crop yield, given yield is > 0, with a gamma, log normal, or other continuous GLM. $\endgroup$
    – RobertF
    Mar 15, 2014 at 13:07
  • $\begingroup$ @RobertF not really enough to do so, and anyway if I were to go that route I might use a tobit model. $\endgroup$ Mar 15, 2014 at 17:32
  • 1
    $\begingroup$ If you are worried about too many fixed effects resulting in biased standard errors, why not use a mixed effects model? $\endgroup$
    – Andrew M
    Nov 12, 2014 at 3:52

2 Answers 2

2
+25
$\begingroup$

I would advise against this practice. Poisson distribution pretty much depends on the scale. Would you multiply by 10, 100, 1000, or 10000? Why? You will get different results when multiplying with different constats! See this simple example:

scale1 <- 5
scale2 <- 100
p1 <- rpois(99999, 100*scale1)/scale1
p2 <- rpois(99999, 100*scale2)/scale2
p2dens <- density(p2)

hist(p1, breaks = 20, freq = FALSE, ylim = range(p2dens$y))
lines(p2dens, col = "red")

enter image description here

You see that with different scales you get completely different distribution!

From this, I would think that poisson should be used for count data only. If you do use it anyway, you definitely need to handle overdispersion (you can try it and look at the residuals) and try different multiplying constants, but I think cleaner approach would be to do normal regression with log(y + c).

$\endgroup$
7
  • $\begingroup$ Negative binomial handles the overdispersion problem. And for log(y+c), how do you choose c? What if results are sensitive to the choice of c? $\endgroup$ Nov 8, 2014 at 18:00
  • $\begingroup$ @ACD, I guess the result will be less sensitive to c than to the multiplying constant you are proposing. 2) are you sure NB handles the problem completely? You wrote that it underestimates SEs. $\endgroup$
    – Tomas
    Nov 8, 2014 at 18:40
  • $\begingroup$ Your example doesn't really represent my data so well. It looks more like a hist(rgamma(1000,1,1)), but with zeros. Also, I don't follow how the multiplication would change things at all -- it's like taking data measured in decimal meters and recasting it as whole centimeters. $\endgroup$ Nov 9, 2014 at 1:10
  • $\begingroup$ And re: se's: that paper that I linked found that umtiplying the variance-covariance matrix by the ratio of the deviance to the degrees of freedom brought coverage to nominal -- though it was a simulation study, and he never says why it works. $\endgroup$ Nov 9, 2014 at 1:15
  • 2
    $\begingroup$ @ACD: "Your example doesn't really represent my data so well." - that's another argument why you shouldn't use poisson! My example shows poisson distribution, which you would assume in your data by using poisson model, and how this poisson model depends on the multiplying constant. The constant changes things a lot! It's because poisson assumes that variance = mean value. $\endgroup$
    – Tomas
    Nov 9, 2014 at 10:02
1
$\begingroup$

I can't recommend your proposal. If $\mu_i$ is the expected value of data point $Y_i$ and $\sigma^2_i$ is its variance, then taking $\tilde Y_i = c Y_i$ gives you a new mean $c \mu_i$ and variance $c^2 \sigma^2_i$. Under the Poisson model, $$\mu_i =\sigma^2_i,$$ hence your proposal is equivalent to assuming a fixed dispersion $c$ in a quasi-Poisson model, since $$c E(\tilde Y_i) = \mbox{Var}(\tilde Y_i).$$ Unless you had good prior information about what $c$ should be, this would be hard to defend, compared to the typical approach for a quasi-Poisson model, which is to estimate $c$ through the mean of the squared Pearson residuals. (For example, R glm(..., family='quasipoisson')). Another recommended approach would be to use a robust "sandwich" estimate.

In any case, your comment about ugly residuals suggests to me that you might have model mis-specification of your mean function $E(Y|X)$, because it is the specification of the mean function that determines what your residuals look like. Assumptions on the mean-variance relationship (which, apart from the default, but non-intrinsic use of the $\log$ link function, is all that the Poisson model buys you) will only change the efficiency of your estimator and the validity of its estimated standard errors. Fixes for mean-model mis-specification could include:

  1. Adding additional covariates (if you've got others measured)
  2. Including interactions or splines/polynomials of covariates
  3. Using a different link function
  4. Transforming $Y_i$ (eg Box-Cox transforms).

For the zero-inflation, you should look into both zero-inflated and zero-altered poisson/negative binomial distributions, available here (for R anyways.) The difference between the two is that zero-inflated assumes that the failures are a mixture of the normal rate of zeros from the underlying distributions, plus an additional, special process that generates zeros, while the zero-altered distributions assume a two-part, sequential model, where an initial "hurdle" needs to be cleared before you get positive counts.

$\endgroup$
2
  • $\begingroup$ OK, I'm convinced that poisson is wrong. But I'm not yet convinced that negative binomial or quasipoisson would be wrong. Would it not be the case that the estimated dispersion parameter would be proportional to $c$? The ugly residuals come from when I add a small constant to lose the zero values before logging -- residuals are otherwise fine. And the data don't really look zero-inflated as such -- no large spike at the y-axis. Any idea whether zero-inflated models are robust to the incidental parameters problem? $\endgroup$ Nov 12, 2014 at 1:18
  • $\begingroup$ The "incidental parameters problem" sounds like another name for the Neyman-Scott problem: nuisance parameters growing with the sample size yields inconsistent estimates. For that, you could try mixed effects, two-stage estimators, or conditional likelihood estimators. The negative-binomial or quasipoisson should be motivated from there being particular mean-variance relationship, eg, you believe that there exists a constant $k$ such that Var$(Y_i) = k E(Y_i)$ (quasi-Poisson) or that Var$(Y_i) = E(Y_i) + E(Y_i)^2/k$ (negative-binomial). $\endgroup$
    – Andrew M
    Nov 12, 2014 at 3:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.