4
$\begingroup$

Given two samples of arrival rate of a process (they should end up something like a Poisson distribution):

  • One over a long period (something like several days)
  • Another from a shorter period time (maybe 30 minutes)

How can I compare the shorter distribution to see if it varies significantly from the longer distribution? Ideally using an existing function in R?

I believe a Q-Q Plot is showing this visually, but I am looking for a single number that would represent how far the shorter sample is from the longer sample.

Also, perhaps better suited for another question. I am a bit curious about how I might say what amount of time would be a representative sample for the longer period of time.

For reference the longer period plot looks something like the following, but it might very somewhat as I decide which attributes I might want to filter out:

enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ (Please be gentile, I am aware of my need to spend more time on the basics :-P ) $\endgroup$
    – Kyle Brandt
    Commented Dec 1, 2011 at 16:27
  • $\begingroup$ Poking at this problem, I think how to normalize the X axis on a non-standard distribution will be a component of this problem. $\endgroup$
    – Kyle Brandt
    Commented Dec 1, 2011 at 19:27

1 Answer 1

Reset to default
3
$\begingroup$

The ks.test (Kolmogorov-Smirnov test) function in R will return a p-value for a two-sample test of the null hypothesis that x and y were drawn from the same continuous distribution, where x and y are would be your two samples. However, if the number of observations in the shorter data set is small, the test will have very little power, i.e., ability to identify situations where the underlying distributions are actually different.

I assume your second question refers to the sample size needed to generate a representative sample? This depends upon the arrival process itself and upon your criterion for "representative." If the arrival process is indeed Poisson, the distribution is characterized by the mean, so the criterion would, in effect, be how accurately you can estimate the mean of the process. Given a rough estimate of the mean and the criterion, you can calculate an estimate of the sample size necessary to achieve whatever accuracy you want.

Maybe you could expand on this question?

Improve this answer
$\endgroup$
10
  • $\begingroup$ For now I guess forget about the representative sample aspect. I basically want to be able to take two these distributions, normalize them in the sense that the actual frequency counts (y-axis) and the values of the x-axis don't matter in an absolute sense, and determine if they are the same shape. (I imagine you want to shoot me for describing this as a picture :-P ). So a number that would say something like "The sample has a dip, or a spike, a long tail, etc. The number wouldn't really have to describe what it is, just an estimate of degree. $\endgroup$
    – Kyle Brandt
    Commented Dec 1, 2011 at 19:35
  • $\begingroup$ In short it would tell me if the sample arrival process is "strange" $\endgroup$
    – Kyle Brandt
    Commented Dec 1, 2011 at 19:37
  • $\begingroup$ Just to clarify: do you have interarrival times or frequency counts of arrivals over time periods of fixed length? And many statisticians use pictures to describe statistical phenomena, so don't worry about it! $\endgroup$
    – jbowman
    Commented Dec 1, 2011 at 20:28
  • $\begingroup$ It "is frequency counts of arrivals over time periods of fixed length", in my case it is per second. $\endgroup$
    – Kyle Brandt
    Commented Dec 1, 2011 at 20:31
  • 1
    $\begingroup$ Thanks for the plot. The Poisson would be visually indistinguishable from a Normal at those high x-axis values. It almost looks like you have a mixture of two distributions, one with a peak around 190 and the other with a peak around 290. A problem with very large sample sizes is that you can almost always reject the null hypothesis (because it's almost always false, in reality) unless you increase the level of the test (e.g., from 95% to 99% to 99.9%) as the sample size increases. Have you tried overlaying plots of the two cumulative density functions? $\endgroup$
    – jbowman
    Commented Dec 1, 2011 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.