2
$\begingroup$

I have a data set from a PC user that is related to the number of user files generated during a 3-year period. It stores how many files per day are created. There are some days when the user doesn't create files, and this is omitted from the data set. Using R, I created a variable with this data using

allday <- c(1,1,1,2,1,1,1,1,1,1,1,11,1,1,1,4,1,1,4,1,1,1,1,1,1,1,1,1,3,2,2,1,3,1,2,
    1,1,1,1,1,1,1,2,3,1,2,3,3,6,5,1,1,1,1,4,2,2,3,1,2,1,4,4,6,6,13,1,5,1,3,5,5,5,5,
    50,10,2,2,9,16,14,6,5,1,4,8,3,3,3,3,15,6,15,3,3,4,14,2,4,4,8,3,3,3,8,9,4,1,7,6,
    6,1,8,4,1,2,27,14,2,9,9,1,7,6,10,4,3,2,1,10,13,9,2,5,3,2,14,8,40,21,17,13,1,4,12,
    13,3,1,10,8,3,7,19,5,6,10,7,6,5,11,11,5,2,11,14,1,5,21,1,6,15,18,19,19,18,3,4,1,
    2,13,1,17,6,4,3,7,11,14,6,24,4,20,3,7,3,24,10,2,3,11,2,14,11,14,28,65,8,22,8,18)

then I get a summary of my data

> summary (alldays)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   1.000   4.000   6.657   9.000  65.000

Then I generated a test data set called "gen"

gen <- rpois(n = 210, lambda = 6.65)

Then I get a summary for the generated data

> summary(gen)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.000   5.000   7.000   6.857   9.000  16.000 

I set

Hypothesis 0 This data are from same distribution

Hypothesis 1 This data are not from same distribution

The histograms are quite different, but when I do a chi square test

> chisq.test(gen, alldays, correct = FALSE)

    Pearson's Chi-squared test

data:  gen and alldays
X-squared = 328.64, df = 405, p-value = 0.9978

I can conclude that H1 is true, so this is not Poisson distribution.

Is this conclusion correct, and maybe should I seek a distribution with higher data dispersion?

$\endgroup$
2
  • $\begingroup$ 1. Please merge your accounts. 2. Why did you remove the zeros? $\endgroup$
    – Glen_b
    Jan 28, 2017 at 0:57
  • $\begingroup$ See the discussion at this previous question of yours on the same sort of problem - choosing a model for the number of files created per day by a user - here: finding distribution of real world data set for prediction, which identifies a better model than the Poisson in that case (and explains why you may not want to remove zeros). However, the model suggested there will not be suitable for this data set -- nor will any other single distribution -- for reasons identified below. $\endgroup$
    – Glen_b
    Jan 28, 2017 at 1:08

1 Answer 1

3
$\begingroup$

There are many issues and errors. In brief I'll say that the entire enterprise is probably misplaced in several different ways at the same time, but I'll address things in some detail. The issue at the end is the most serious in terms of producing a meaningful description of the process - it renders all the earlier concerns I mention somewhat moot.

  1. If you want to identify the distribution of number of files generated, it's not clear why you would remove the zeros; they're part of the process. [Even if you need to consider say a 0-inflated distribution or a hurdle model to adequately describe the zeroes, I wouldn't just eliminate them without a very good reason.]

  2. It's not a good idea to test if something is well approximated by a Poisson by sampling from a Poisson -- that introduces unnecessary noise to the testing problem (though it can be a useful diagnostic if you general many such samples). Even if you were to do that, you'd also have to include code to eliminate 0's from the Poisson that was generated because you need to compare with a truncated Poisson, since you truncated your data (you may not have got 0's in your sample but that was pure luck; if you do it several times you'll get some sets with 0's).

    You would instead compare the data with the expected frequencies under a Poisson model (as we'll see later). However, your estimate of the Poisson parameter was based on the truncated data and so that estimate is incorrect - it doesn't account for the effect of leaving out the zeros.

    [Note also that your code is wrong -- the variable you created (allday) is not called the same name as the one you tested (alldays)]

  3. Those considerations aside, you also did the test you attempted to do wrong. You'd need to tabulate the counts (count how many days had "1", how many had "2" and so on) and then perform a chi-squared test on those tabulated values -- but that would still ignore the fact that you estimated a parameter to generate your sample, so your degrees of freedom wouldn't apply.

    But even if you fix all of those things, that's not the right approach to test goodness of fit ...

  4. If you want to do an actual goodness of fit test, you could set up the expected and observed counts and calculate a chi-squared goodness of fit test. The d.f. would be the number of categories in your table minus 2. You'd lose one degree of freedom because the total expected would equal the total observed and you'd lose one degree of freedom for the estimation of the Poisson parameter.

  5. However, if you were to test goodness of fit, you probably don't want to do that test -- a chi-squared test will tend to have low power against smooth deviations from the model because it ignores the ordering in categories.

  6. On the other hand, you likely shouldn't do a goodness of fit test in the first place -- you can be confident even without looking that your process won't be exactly Poisson, so a test is answering the wrong question.

  7. All the caveats I offered on your previous question with the smaller data set (such as dependence over time and heterogeneity) still apply.

    All that said, you can see by direct inspection that this can't be Poisson (the variability is way too large relative to the mean), so it'd be a waste of time trying to test it even if it made sense to do that.

  8. In your previous question I suggested a negative binomial model. That would be a better choice than the Poisson, but it's not necessarily going to be a good fit for this data; it looks to me like there's considerable time-related heterogeneity -- indeed you can see clear growth

    plot of allday against observation index

    As such, it makes no sense to look at the marginal distribution, since the distribution is changing over time. If you're trying to model the future, you cannot reasonably ignore the trend. The changing distribution will make the marginal distribution seem to be high-variance (and quite skew) even if the conditional distributions are only mildly skew with moderate variance.

    (it's not easy to asses any likely serial dependence around that growth trend when you omit the zeros, but the potential existence of serial dependence is a likely possibility that should be kept firmly in mind for any future model).

    It may be that a Poisson or a negative binomial model that accounts for growth (perhaps some form of time-series / regression model) could work reasonably well for this data but we'd need to remember that at best any such model will be an approximate description; we would not expect that with more data we could not find that it didn't fully describe the process (for all that it might prove a useful model in some contexts).

$\endgroup$
1
  • $\begingroup$ I find this answer both hilarious and accurate. If I had a registered account I would upvote. $\endgroup$
    – gammer
    Jan 28, 2017 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.