1
$\begingroup$

Can I apply the Poisson distribution on the following type of data set?

I have two types of processes, each generating events. The actual data that I posses are sets of these timestamps.

The occurrences are independent and random.

Unfortunately, events are clustered because the process generating them pauses for random periods (short AND long periods of pause). It is not the pause periods that interest me, but rather the periods during which the process is on (therefore the pause periods of time should not influence my analysis).

Since the data is not uniformly distributed over time, but rather comes in clumps, I am not sure whether I can use the Poisson distribution here.

My ultimate goal is to analyze the difference in frequency between two variants of such processes. I will compute the probability distribution for each process, for the interval of a day and see how the two differ. My fear is that the pause intervals would skew the distribution so badly that they will become meaningless to my goals.

The following are histograms for the timestamps, each bin representing one day: https://dl.dropboxusercontent.com/u/17450714/i-1.jpg

https://dl.dropboxusercontent.com/u/17450714/i-0.jpg

If Poisson is not good for this data, how may I analyze it by excluding the huge pause periods? Cluster analysis?

Thanks

$\endgroup$

1 Answer 1

6
$\begingroup$

In order for the Poisson distribution to be the limiting distribution of a set events, the events need to come from a Poisson process. This means that (forgive me if I am less precise than I need to be, my trusty copy of DeGroot is at work):

  1. In two disjoint intervals of time, the number of events are independent.
  2. The probability of two or more events in a small interval is of sufficient order of magnitude smaller than the probability of one event as to be effectively $0$.
  3. The probability of the number of events in an interval is solely a function of the size of that interval

The data does not have to be distributed uniformly over time as long as the probability of the events depends solely on the size of said interval (imagine a Poisson with a small $\lambda$). In your case, it is probably true that in the “clumps” the process exhibits Poisson-like behavior, but it is not true for the time between those clumps.

Perhaps you can "snip out" the gaps and work with the intervals in which events occur.

$\endgroup$
4
  • 2
    $\begingroup$ +1 Only conditions (1) and (2) are needed: see en.wikipedia.org/wiki/Poisson_process#Characterisation. Condition (3) is tantamount to homogeneity. These data evidently are not homogeneous (according to the second graphic). $\endgroup$
    – whuber
    Aug 29, 2013 at 20:17
  • $\begingroup$ So then I can apply the Poisson distribution even though there is no homogeneity in the data? That's what the wiki article suggests $\endgroup$
    – cdmihai
    Aug 29, 2013 at 20:54
  • 2
    $\begingroup$ No, because I do not think you are dealing with the case that your Poisson parameter is a function of time. For all you know it is continuous, but you are simply missing observations (like a train passing in front of the window doesn't mean what is behind it stopped). Which is why I think you are better off analyzing the data you have over the timestamps you have. Dropbox is blocked here, but shave off a bit from either end of the non-gaps, to prevent distortions coming from the return to visibility, and analyze rate-per-VISIBLE time. $\endgroup$
    – Avraham
    Aug 29, 2013 at 21:49
  • 1
    $\begingroup$ In the above, where I said "continuous" I meant "constant;" sorry. $\endgroup$
    – Avraham
    Aug 30, 2013 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.