Equivalent to K-S test on discrete data with uneven quantization

Question

I have some data resulting from a simulation that consists of several groups, each containing a single real datapoint and a variable number of matched controls. I take the rank of each real value within its distribution of controls, and normalize the ranks to between 0 and 1.

Now I would like to test whether, across all groups, the ranks of my real datapoints are different to what I would expect by chance. However, I'm unsure about how to do this, since the distribution of possible normalized ranks across groups is discrete with uneven quantization (due to the variable number of control datapoints in each group).

I expect the CDF for the null distribution to look a bit like this:

I have considered the possibility of bootstrapping a null distribution by drawing randomly from each set of possible normalized ranks, then doing a 2-sample K-S test on the real and null distributions. Is this the way to go, or would there be a more appropriate test?

Edit: I've also posted a more complete description of the problem I'm trying to solve here.

Answer 1

This is a really interesting question.

I think the simulation makes sense, but I'd look at it from the point of view of a randomization test, though in this case it's essentially the same as a bootstrap test; they're both resampling tests and once you go to ranks, they work essentially the same.

In effect, under the null, and conditional on the group numbers, within each group you randomly allocate the label 'treatment' to one of the set of combined "treatment + matched control" ranks.

You could then compute a 2-sample KS statistic for each such "resample" -- but you could compute an A-D test, or any number of other such tests, just as easily.

That process gives you the null distribution of your statistic, which you compare your original sample statistic with.

So yeah, I think your basic idea is sound.