1
$\begingroup$

I am trying to predict the number of events happening with the next time window, given the current values of some input variables. I am trying to pick up a good family of distributions to describe this data. Given the nature of the problem, Poisson distribution seems like a good idea.

When I plot histograms of data for different values of input variables, I see that there's a lot of mass at 0 (around 90-95% if I don't condition on anything), and the distribution for non-zero values looks like an exponential: the probability mass function is gradually decreasing. That already makes the Poisson distribution assumption questionable.

Furthermore, for the Poisson distribution the mean equals variance. In my case, however, if I plot the mean and variance for different values of conditional variables, I see a linear relation between them: variance = const * mean, where const is very high. Hence, again, Poisson distribution does not seem to be a good choice to fit this data. Which family of distributions would you suggest?

Improve this question
$\endgroup$
6
  • $\begingroup$ Would the gamma distribution work for this - mean = $k\theta$ and variance = $k\theta^2$? i.e. mean = variance * $\theta$. $\theta$ is the scale parameter and $k$ is the shape. $\endgroup$
    – Eumenedies
    Commented Jun 1, 2017 at 9:17
  • 4
    $\begingroup$ Your question is not clear. For one thing, exponential distribution is for positive data. Next, see en.wikipedia.org/wiki/Exponential_distribution for the exponential distribution mean is not equal to variance. Maybe you have some other distribution in mind? It would be better if you reformulated, explaining your practical situation, what measurements do your variable represent, without using statistical jargon. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 1, 2017 at 9:18
  • $\begingroup$ @kjetilbhalvorsen: hope it is better now $\endgroup$
    – Ulysses
    Commented Jun 1, 2017 at 10:07
  • $\begingroup$ Do you have count data? that is, all the observations are nonnegative integers. With so many zero counts, you could look into zero-inflated models (search this site). With variance=const*mean, poisson distribution will not fit, but poisson regression can still be used, with corrections for overdispersion. See for example stats.stackexchange.com/questions/20826/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 1, 2017 at 10:57
  • 1
    $\begingroup$ Then I would start out with (possibly zero-inflated) and overdispersed Poisson regression. See stats.stackexchange.com/questions/45262/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 1, 2017 at 12:12

1 Answer 1

Reset to default
2
$\begingroup$

You have count data, that is, the observations are non-negative integers which can be modelled as some kind of counting process. The high proportion of zeros can be taken care of by some kind of zero-inflated model, see for example Zero-inflated count models in R: what is the real advantage?

You also say that $\text{variance}=\text{const} \cdot \text{mean}$ with const much larger than one. That is a situation modeled perfectly by poisson regression, but with some correction for overdispersion. For any glm (generalized linear model) the mean-variance relationship is fundamental to efficient estimation, since this is what determines the weight used in the IRLS (iteratively reweighted least squares, see Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?) estimation algorithm. But weights are determined by the relative values of the variance, not the absolute values. So proportionality to the mean, not equality, is sufficient for the poisson model to give optimal weights. But then, the standard errors will be wrong, which is what is corrected for by overdispersion corrections. There is a lot of posts here about poisson regression and overdispersion, so just search the site.

Another possibility is to use a negative binomial distribution family. That is also covered by a lot of posts, see for instance Difference between binomial, negative binomial and Poisson regression

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks, do I understand correctly: you suggest using both zero-inflated models and overdispersion methods? $\endgroup$
    – Ulysses
    Commented Jun 5, 2017 at 11:42
  • $\begingroup$ Yes. You should at least investigate that possibility! $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 6, 2017 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.