Finding distribution of real world data set for prediction

Question

I have a question regarding a data set. I have observations about the number of files generated by a user over two months.

My data set is

1,1,1,1,1,1,1,6,1,3,2,4,5,5,9,5,8,15,14,4,3,1,1,2,9,4,13,
3,13,3,5,6,11,1,18,18,2,17,11,4,3,3,14

-- a total of 43 days with observations (two months = 60 days, but the user didn't create any file during 17 of these days). Do I need to include the days with 0 files created in the data set and set their value to zero?.

Summary statistics show that

min:  1   max:  18 
median:  4 
mean:  5.883721 
estimated sd:  5.310409 
estimated skewness:  1.031521 
estimated kurtosis:  2.81521

 Min.  1st Qu. Median    Mean  3rd Qu.   Max. 
1.000   1.500   4.000   5.884   9.000  18.000 

Now I am trying to find a possible distribution.

I want to make some prediction of how the user will continue to generate files (network simulation).

When I do a Cullen and Frey graph, my observation is placed in the gray area (denoting a Beta distribution). That confuses me since I read that this is typically for values between 0 and 1.

Any suggestions and guidelines are welcome since I am quite stuck

Answer 1

the user didn't create any files during 17 of these days. Do I need to include these days in the data set and set the value to zero

Probably, but it depends on what you are trying to achieve. If you're trying to predict what happens on any day, then "no files were created" is a possibility and you should model that. If you're trying to predict what happens only on the days when at least one file is created, then you would omit the zero-files days.

When I do a Cullen and Frey graph, my observation is placed in the gray area (denoting a Beta distribution). That confuses me since I read that this is typically for values between 0 and 1.

1. Since the graph uses squared skewness and kurtosis as a basis to suggest distributions, the distributions considered may be arbitrarily multiplied by a scale factor (of either sign) and shifted (since that doesn't change skewness or kurtosis).

   So if you're looking at a beta distribution you're really looking at a four-parameter beta model, not the two parameter beta.

2. The "Cullen and Frey" graph is basically just an old Pearson plot with some bootstrap details added.

   As such, most of the distributions listed are Pearson-family, with a few completely-arbitrarily chosen ring-ins added on top. All of them are continuous.

   You have discrete data which mostly takes small non-negative values (75% of values are 0-5). In general you should not try to approximate small non-negative integers by a continuous distribution. (Even if you did have continuous data, I would be very cautious about using the Pearson plot in general as a tool for choosing distributions.)

3. It's unlikely any single simple-form distribution will provide a good fit to the distribution of number of files (though with so little data you probably can't rule out some simple distributions); this is because the process is likely a mixture of different things (i.e. if you had a lot of data you'd probably need some some kind of mixture distribution but even the components wouldn't necessarily fit any simple distribution very well).

4. It is possible to generate predictions without assuming any single distributional form; it's worth keeping the possibility of some nonparametric approaches in mind.

5. If you're doing prediction you should probably consider the possibility of using predictor variables (IVs) -- such as day of week effects for example - and even of serial-dependence (e.g. if we did a lot of files yesterday maybe there's a good chance we'll have more than average tomorrow). There may also be trends over time. Such things may lead to somewhat more useful predictions, especially if you can get more data. (If you're going to look at doing this kind of thing, you really need the zeros in there.)

6. If you're determined to fit a single distribution in any case, I'd suggest an initially reasonable approximation would be had from the negative binomial distribution. Supplementing your data with the zeros, we have:

That looks like it might be adequate as a basis for very simple applications.

However, note that we have not yet considered the fact that - even if our model were correct (that the data were generated as a set of iid negative binomials) - the parameter estimates are uncertain, but we haven't yet incorporated that. As a result our fit is closer to the data (in a particular sense) than it is to the notional population from which the data were drawn. If we were interested in something like a prediction interval we would need to incorporate that parameter uncertainty into our interval (there are ways to do this at least approxmiately). Ideally we'd also want to incorporate the fact that the choice of distributional model was itself uncertain -- there's a variety of roughly-as-plausible distributional choices that are also reasonably simple, and that uncertainty about which to use also leads to some uncertainty in prediction intervals (the effect of model-specification uncertainty/model selection uncertainty). It is possible to incorporate at least some of that into prediction intervals as well, though this kind of uncertainty is often ignored. If you have other purposes in mind this may be less important to worry about -- note that to some extent regularization (such as shrinkage) and incorporation of parameter uncertainty will tend to pull in opposite directions.]

Answer 2

Including zeros or not depend on what you are studying. If your question is "how many files are created provided that some file is created that day", then you shouldn't include zeros, but if your question is a more general "how many files are created a day", then zero is a possible answer to that question and zeroes should be included in the dataset.

Furthermore, the most usual discrete random distributions (Poisson, binomial...) yield zero values, and therefore including zeroes can help you to fit a known distribution.

Answer 3

Was it 17 consecutive days that data was not collected ? I no then I would linearly interpolate (while holding my breath !) to get estimates of these 17 missing values. I would then identify a useful model which I would then use to make a prediction. I would use re-sampling of model errors and enable pulses to be incorporated when generating confidence limits. The fact that the data is not continuous is not a real concern to me only a slight bother as reasonable models can be constructed for integer data just as long as the values are not too repetitive. Recall that the airline time series is discrete data. If you can interpolate please do so.