Equivalent to K-S test on discrete data with uneven quantization

I have some data resulting from a simulation that consists of several groups, each containing a single real datapoint and a variable number of matched controls. I take the rank of each real value within its distribution of controls, and normalize the ranks to between 0 and 1.

Now I would like to test whether, across all groups, the ranks of my real datapoints are different to what I would expect by chance. However, I'm unsure about how to do this, since the distribution of possible normalized ranks across groups is discrete with uneven quantization (due to the variable number of control datapoints in each group).

I expect the CDF for the null distribution to look a bit like this:

I have considered the possibility of bootstrapping a null distribution by drawing randomly from each set of possible normalized ranks, then doing a 2-sample K-S test on the real and null distributions. Is this the way to go, or would there be a more appropriate test?

Edit: I've also posted a more complete description of the problem I'm trying to solve here.

• I'd have been inclined to use the tag hypothesis-testing rather than one of distributions or empirical here. No great matter though. – Glen_b Apr 12 '13 at 0:19
• That makes more sense - I've re-tagged accordingly – ali_m Apr 12 '13 at 15:02

This is a really interesting question.

I think the simulation makes sense, but I'd look at it from the point of view of a randomization test, though in this case it's essentially the same as a bootstrap test; they're both resampling tests and once you go to ranks, they work essentially the same.

In effect, under the null, and conditional on the group numbers, within each group you randomly allocate the label 'treatment' to one of the set of combined "treatment + matched control" ranks.

You could then compute a 2-sample KS statistic for each such "resample" -- but you could compute an A-D test, or any number of other such tests, just as easily.

That process gives you the null distribution of your statistic, which you compare your original sample statistic with.

So yeah, I think your basic idea is sound.

• Thanks for your response - it's nice to have my sanity confirmed every now and then! What I'm doing for the moment is quite similar to what you propose, but in my case it wouldn't make much sense to compute the K-S or A-D test statistics within a group, since each group contains only one 'treatment' datapoint. Instead, I just compute the sum of the normalized ranks across groups as my test statistic and bootstrap by randomly reassigning the 'treatment' label within each group on each iteration. – ali_m Apr 12 '13 at 10:48
• I think I might post a more complete description of the problem I'm trying to solve as a separate question, since I still feel like my whole approach is almost certainly overly conservative. – ali_m Apr 12 '13 at 10:50