# Alternative to Kolmogorov-Smirnov test when parameters are estimated from the data

I need to compare whether two distributions are similar when the values are scaled by the mean of each of the distribution. One limitation of ks-test as per http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm is "if location, scale, and shape parameters are estimated from the data, the critical region of the K-S test is no longer valid."

Consider for example:

Data1 consist of 10000 numbers from uniform random distribution [0,1] with mean 0.5

Data2 consist of 10000 numbers from uniform random distribution [0,10] with mean 5.001

If I compare Data1 with Data2/10 then ks-test gives that both the distribution are same; while comparing Data1/0.5 with Data2/5.001 gives that the distribution are different. Is there a way to check the similarity between the distributions in such cases?

Edit: As the answer suggests I can use ks-test where the p-value is determined via permutation.

My additional difficulty is that the data-points are integers:

Data1 consist of 10000 integers from uniform random distribution [0,10] with mean 5

Data2 consist of 10000 integers from uniform random distribution [0,100] with mean 50.001

Is there a test to compare whether Data1 and Data2 are similar apart from the scale? Further, I do not know the actual scale and I am determining it from the data.

These examples are just a proxy for my actual data, which are two experiment where 10000 people rated a movie on a scale [0,10], while in other case 10000 different people rated the same movie on a scale [0,100]. I want to check apart from the scale can one say that whether the distributions are same or not.

• Re the edit: the underlying distributions are prima facie different: one is supported at 11 values and the other at 101 values. The data will reveal this with astronomical certainty because the chance that the support of Data2 will include more than 11 values is greater than $1 - 10^{-9615}$. In fact, "scale" makes no sense here because the two distributions you name are not scaled versions of one another. – whuber Jul 4 '13 at 15:40
• Even if Data2 only have integers that are multiple of 10, i.e., 0,10,20,...100, thus allowing only 11 possible values, the distribution are still dissimilar. – imsc Jul 4 '13 at 16:24

One option is to still use the KS test statistic, but instead of using the standard p-value from the KS test (which as you say is not appropriate when estimating from the data), calculate the p-value using a permutation test. The basic steps would be:

Calculate the KS test statistic for the data as is (divided by the estimates).

Now combine the 2 datasets (already divided) and randomly split them into 2 sets of 10,000 (or whatever the original sample size was) and compute the KS test for these new "samples".

Repeat the above many times (999, or 9,999).

The p-value is the proportion of test statistics that are as extreme or more extreme than your original test statistic.

• +1, just out of curiosity, why the strange number of iterations? I usually use B = 10k. – gung - Reinstate Monica Jul 3 '13 at 22:01
• @gung My guess: because then the usual quantiles people are interested in will correspond to observations rather than needing interpolation. – Glen_b -Reinstate Monica Jul 3 '13 at 22:23
• Many thanks, I checked and it worked for the example in question. I was looking for such a permutation test before but could not find any. Do you have any reference. Additionally, my actual data points are integers, where this above method does not seems to work. Do you have any suggestion in this case. – imsc Jul 4 '13 at 8:04
• @gung, The original data provides one permutation, so adding an additional 9,999 random permutations gives a total of 10k and p-values, etc. turn out nicer when using a power of 10 (and using base 10 arithmetic). – Greg Snow Jul 5 '13 at 17:04
• @gung, Your method could lead to a p-value = 0, but since there is always at least one permutation (the original one) that is as extreme as the data, the p-value should always be at least slightly higher than 0. Including the original guarantees this. – Greg Snow Jul 5 '13 at 17:50

I can't make much sense of the question on any level. The detail about the values being integers is really important!

1. It appears that you are using a random number generator for uniform integers with different ranges (else how do you know that your distribution is uniform?). When you sample from a distribution on [0, 10] then there are 11 possible values and the mean of the distribution is necessarily 5. This is not a case where you are estimating the mean from sample data; you know the mean. Similarly, the distribution on [0, 100] has 101 possible values and the mean of such a distribution is 50. Extending this, the usual general advice about location, scale and shape parameters and Kolmogorov-Smirnov tests is irrelevant, because the range of a uniform distribution defines the distribution uniquely.

2. If you scale by the mean of each distribution, then the possible values are 0, 0.2, ..., 1.8, 2 in one case and 0, 0.02, ..., 1.98, 2 which in principle are not exactly the same distribution. With a large enough sample size, a Kolmogorov-Smirnov test might detect this, but the problem is that K-S test can't tell what fraction of any discrepancy is (a) sampling variation (b) the difference in domain of the distributions (c) something else.

3. If you used a random number generator then your question boils down (other than the question of different domains) to whether the generator is any good and you will need better and different tests to find flaws in the random number generator of any respectable statistical software.

4. It's possible that you just have data on 0, 1, ..., 9, 10 and 0, 1, ..., 99, 100 and your hypothesis is that both distributions are uniform, in which case problem #2 still bites. I'd still argue that the means are part of the hypothesis and don't need to be estimated from the data.

A personal prejudice is that the Kolmogorov-Smirnov test is overrated any way. If it reveals a problem, you need other methods to find out what the problem is. With 10,000 data points important discrepancies will show up on a quantile-quantile plot of the two distributions and/or quantile plots of each separately. Alternatively, the histograms should clearly be flat and that is an easy thing to try.

How well the K-S test works with discrete distributions is also a crucial question. Others will be deeper into the literature and theory, but my instinct is that discreteness doesn't help.

• Many thanks for your comment. I should have crafted the question more carefully. Indeed my data do not come from random uniform distribution, but I was using the examples to shown even when the distribution are known, I could not find a way to say that they are statistically the same. So my case is your #4, and as you have said my data suffer from #2. I do not have any linking for KS test, I just could not find any better statistical alternative. I indeed plotted them and most of the distribution overlap, but I want to say whether the discrepancy is statistically significant or not. – imsc Jul 4 '13 at 9:00
• I see. However, if the examples aren't really typical, or contain details that don't really apply, it becomes even harder to see what the question is. The bottom line is that you seek a precise answer to a very fuzzy question. Do distributions on different domains differ? has to be firmed up in some way. – Nick Cox Jul 4 '13 at 9:20
• The detail on movie scores just added makes your problem much more vivid. I think it's the kind of detail most people answering questions would most want to see, and is not at all incidental. – Nick Cox Jul 4 '13 at 9:27

It may be possible that a distribution-free test like the Wilcoxon-Mann-Whitney, which is a test based on rank, not value. In your example, multiplying all the values from Data1 will not change their respective iinternal ranks, but now they will be on the same "scale" as Data2. As such, the MWW test may provide you with insight as to whether or not "rescaled" Data1 is "different" from Data2. Of course there will be "clusters" in Data1 and probably none in Data2, in which case you may want to consider some type of kernel smoothing.