1
$\begingroup$

I am trying to see which model fits better to an email count data.Tried to fit the count data to Poisson using chi-square test. Did not fit. Tried to fit the count data to Negative binomial by estimating the parameter, and using the chi-square test. Did not fit, although it reduced the chi-square statistic than that before. What could be the error ? and how should i proceed ?

$\endgroup$
1
$\begingroup$

Although the question is a bit vague, I will try to give some information. Frequency is more often modeled by:

  1. Poisson
  2. Negative Binomial
  3. Binomial

All these distributions are determined by their mean and variance. If

$$\newcommand{\Var}{{\rm Var}}\frac{\Var[N]}{E[N]}\approx 1$$

you should choose Poisson where $E[N]=\Var[N]=\lambda$ .

If

$$\frac{\Var[N]}{E[N]}>1$$

you should choose Negative Binomial where $E[N]=\alpha\beta$ and $\Var[N]=\alpha\beta(1+\beta)$.

If

$$\frac{\Var[N]}{E[N]}<1$$

you should choose Binomial where $E[N]=np$ and $\Var[N]=np(1-p)$.

The choice is not strictly mathematical. Parameters can be calculated by solving the equations. If you have enough data, you could calibrate parameters by calculating mean and variance for different periods. The Pearson chi-squared test is used for testing fitting performance but goodness of fit tests in general provide only indications and should not be your only criterion. Empirical vs theoretical distribution graphs could also help to determine fitting appropriateness. You could also try to fit distributions with more parameters, like Conway-Maxwell-Poisson distribution but beware of over-fitting. In case there are a lot of zeros you should see this article.

$\endgroup$
  • 3
    $\begingroup$ I believe this advice will not solve the original problem, since the OP said the negative binomial was better than the Poisson but both were inadequate; there's an implication (and it's a common situation in practice) is that the data are likely even further the negative binomial side of the Poisson (heavier tailed and perhaps more peaked); while the binomial is in apparently the wrong direction (even less of a right tail than the Poisson); so the two obvious choices of the three you mention are explicitly ruled out in the question, and the third by implication. It may be there's a lot of 0's. $\endgroup$ – Glen_b -Reinstate Monica Feb 17 '17 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.