I want to compare the delays of packets belonging to some kind of traffic to the delays of packets belonging to some other (different and larger) traffic, all generated from the same machine.

I would like to see if the distributions of the two groups of delays are the same. Samples for the first group are between size 20 and 2 000 (bust most of the time between 20 and 75), while the comparison group is always of the order of a few thousands (up to 15 000).

When the first group has more than 250 values, I just (randomly) subsample it in smaller groups of size 50, since it I noticed that it really takes nothing for large samples to get rejected with Kolmogorov-Smirnov. If the majority of subsamples gets accepted by the test, the original sample also does.

Now, due to the precision of timestamping, there are lots of duplicate delays (i.e. ties) in my observations. For instance, out of 15000 delays in my second group, I have only 1000 unique values, with a mean of 0.7 milliseconds and a standard deviation of around 7 milliseconds. 200 values appear up to 10 times; 90 values between 11 and 100; 30 between 101 and 300; 4 between 301 and 500.

What should I take into account in order to choose a non-parametric test that best suits my case?

Also, is my subsampling correct?

I've been using a significance level of 0.01, but I'm getting 20% of false positives, of which roughly half greater, half smaller than the second group (after the two-sided test, I always used the two one-sided ones just to check).

Any ideas?


If you want to compare the distribution shapes, you might use Kolmogorov-Smirnov. I didn't quite understand: that if one sample size is 20 and the other one is thousands? Please be clear in your question. Explain your study more clearly please. What is timestamping? And what are these duplicates? What are your study goals? What are your variables? How many groups do you have?

Is your case like comparing packet delay of for example TCP with UDP? So here what are duplicates, what group has thousands of data? What group(s) has (have) only 20 data? By 20 data you mean 20 packets? Or 20 attempts of connection, each with thousands of data? etc...

The point is that if your group has thousands of observations, you should adjust your alpha because at a usual alpha = 0.05, a high number of observations will cause a false positive result with the slightest differences in the two distributions.

  • $\begingroup$ Thanks for your answer! I added more details in my original question. Could you please have a look at it again? Thanks a lot! $\endgroup$ – Ricky Robinson Jun 6 '13 at 19:57
  • $\begingroup$ By duplicates I just mean that certain delay values appear more than once in the same group. $\endgroup$ – Ricky Robinson Jun 6 '13 at 19:59
  • $\begingroup$ Thanks for clarifications. Sorry I couldn't understand any of the technical ones! Just got subsampling and the overpowered KS test. From what I understood, it seems you already are on the right path regarding the statistics. KS is proper for distribution comparison (although there are similar tests as well like Cramér-von Mises). Besides, sometimes the alpha should be set at a very small level (such as 0.001, 0.0001, 0.00001 which is common in pharmacology). It depends on your test power. Too much power needs to be offset by a smaller alpha. $\endgroup$ – Vic Jun 6 '13 at 20:33
  • $\begingroup$ Another way is to subjectively compare two histograms and judge their shapes. (It might sound ironic but it is possible, especially when the sample is huge). $\endgroup$ – Vic Jun 6 '13 at 20:34
  • 1
    $\begingroup$ yes and also z test. For correctly adjusting the alpha (something which should be Always done but is ignored vastly) you should calculate the power of your statistical test. Google "power calculation for kruskall-wallis test" (or similar terms). A part of the formula for this purpose is the alpha itself. So knowing the other parameters (sample size etc.) you can compute proper alpha. $\endgroup$ – Vic Jun 7 '13 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.