Implementing a Kolmogorov-Smirnov permutation test

I would like some advice on how to implement a Kolmogorov-Smirnov test.

This is a condensed version of my previous question -- any further detail can be provided if needed.

I have data from n experimental trials. Each set consists of two paired time-series signals, for which I am calculating an informative "overlap" measure (a single summary value), based on some threshold value. I calculate this summary value for all n sets of data.

I would like to determine whether the distribution of these overlap values is non-random. The distribution of the data is not normal, and having read some details about different comparative tests of distributions, the Kolmogorov-Smirnov test seems most appropriate.

I'm hoping someone can give me advice on how to perform a permutation test using these data. My understanding is that this requires comparing the distribution of the actual data with those obtained from random combinations of the data, however, I'm uncertain of the procedure for implementing and interpreting the result. Any help would be much appreciated.

• To do a Kolmogorov-Smirnov test you typically need a known reference distribution to compare against. If the overlap values were random, what would their distribution look like? Jul 2, 2013 at 16:44
• Randomising the overlap values produces something that looks quite different to the actual data (I'm not sure that the data fit an archetypal distribution particularly well). My thoughts were that I may be able to do a two-sample KS test, comparing the actual overlaps distribution with randomised ones. I could repeat my randomisation a number of times to collect many KS statistics; I'm not particularly confident however that this is a statistically robust way to proceed. Jul 2, 2013 at 22:28
• If your time series were independent, what would the overlap distribution look like? Would it have a mean of zero and be symmetric, for example? Jul 2, 2013 at 23:25
• No - the independent (i.e. randomised) overlap data are highly skewed, if anything like an exponential distribution. The actual overlaps distribution (of the paired time series) does not look like this, however. If anything it is more uniform. Jul 3, 2013 at 11:38
• You are describing what you are observing. I am asking what would the overlap data look like if the two time series were in some sense "equivalent." That is, they were independent, equally accurate, and with the same variance. If you can find the overlap distribution for that case, you have the reference distribution to use for the Kolmogorov-Smirnov test. Jul 3, 2013 at 13:40