5
$\begingroup$

I measured count data across different sites for multiple dependent variables. I encountered a problem with zero inflation, so I added a count of '1' to all my data points, converted all count data to relative proportions (densities), and arcsine transformed my entire data set.

Graphically, it appeared that my data follows a poisson distribution. Consequently, I measured the mean and the standard deviation (not variance), and found that they are approximately equal for some dependent variables, but not all of them. To get an idea about how different the mean is from the standard deviation, I measured the ratio of mean: standard deviation, and the results ranged from 0.29 to 1.34, with an average of 0.81.

My question is: How important is the assumption of mean = variance for analyses of data based on poisson-distribution generating transformations?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ The expected value of Poisson distributed random variable is not equal to it’s standard deviation. It’s equal to it’s variance. So what does your calculations show if you correct this? $\endgroup$
    – Harto Saarinen
    Commented May 10, 2018 at 17:29
  • $\begingroup$ How did you find there was 0-inflation? $\endgroup$
    – AdamO
    Commented May 10, 2018 at 17:54
  • $\begingroup$ Dividing the mean by the variance (which now has much lesser values obviously) gives proportions much higher than 1. Now they range from 3.9 to 36.. In other words, they're not equal at all. Is there anymore info I could provide to help people help me determine the best fitting distribution? I was following advice found here (see user leonbloy): stats.stackexchange.com/questions/26815/… $\endgroup$
    – ziab_m
    Commented May 10, 2018 at 17:55
  • 1
    $\begingroup$ I was concerned you'd say that. This is a typical mistake: this doesn't necessarily mean there's 0 inflation. You can have a lot of zeros in a Poisson sample. $\endgroup$
    – AdamO
    Commented May 10, 2018 at 18:26
  • 1
    $\begingroup$ Read this thread. Unequal mean/variance doesn't concern me. Sounds like you should be fitting a quasipoisson model at the least. But it all depends on the point of the model... I mostly do inference, never prediction. $\endgroup$
    – AdamO
    Commented May 10, 2018 at 19:00

2 Answers 2

Reset to default
5
$\begingroup$

Having "a lot of zeroes" in a dataset does not mean that it is zero-inflated. If the Poisson rate parameter is 0, then 100% of observations are 0 but it is not zero-inflated. Zero-inflated Poisson processes are a special type of mixture distribution that will (usually) give rise to a bimodal frequency distribution.

If a distribution is in fact a zero-inflated Poisson distribution, then it is a corollary that the mean-variance relationship of a Poisson distribution will be violated. That is not the only problem to consider, however, whether doing either prediction or inference. On the other hand, the mean-variance assumption can be violated without zero-inflation. Examples of this are over/under dispersion or other Poisson mixture distributions.

The "importance" of the mean-variance assumption verges on being an opinion-based question. But I guess that is in and of itself an answer to your question. It could be unimportant: The variance might be "only kind of" different from the mean. In that case, any parametric prediction or inference using an incorrect Poisson assumption will mostly agree with the more sophisticated zero-inflated models. On the other hand, count data are often far from "only kind of" in terms of this discrepancy.

Under- and over-dispersion of large magnitude can cause massive miscalibration of error estimates for prediction and confidence intervals (PIs and CIs respectively). In this case, quasipoisson or GEE models correct the CIs and negative binomial models correct the CIs and PIs.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you Adam, that was a very useful response. Sorry about the late response, I had to research all those concepts. $\endgroup$
    – ziab_m
    Commented May 15, 2018 at 14:28
1
$\begingroup$

"converted all count data to relative proportions (densities), and arcsine transformed my entire data set"

This will transform the variance and mean by different ratio's and is a likely cause that you get such high numbers for the ratio 'mean:variance' as between 3.9 and 36 (unless you have good reasons that this high amount of under dispersion might occur, adding 1 extra counts to all of your data, for instance, doesn't do good either and is a source of under-dispersion)

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm currently looking to see if I can choose a completely different distribution to fit my data (e.g. negative binomial), and compare fit that to the poisson distribution. Under this approach, I won't have to transform any counts, or add any pseudocounts. I will have to compare model fits, but I'm still new to all of this so I thank you for the input on the transformation consequences on means and variances. $\endgroup$
    – ziab_m
    Commented Jun 8, 2018 at 22:50
  • $\begingroup$ I think you should start with your assumptions about the process that generates your data. If this is best described as a homogeneous Poisson process, then why do all the tricks with transforming the data? How is your process of choosing the distribution to fit the data (trial and error, or based on some theoretical ideas)? (it might turn into too extreme data massaging, unless you do some verification on new fresh data) $\endgroup$
    – Sextus Empiricus
    Commented Jun 8, 2018 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.