Two-sample permutation Kolmogorov-Smirnov tests

While it is easier to use the Pearson chi-square/Cressie-Read type test, I would like to test the equality of proportions in $k$ categories across two groups using a Kolmogorov-Smirnov type test of the form proposed by Pettitt & Stephens (1977) (see also here).

In particular as the authors of that paper point out, it may have some power against trending alternatives. So their one-sample nominal/categorical Kolmogorov-Smirnov test has the form: $$D_n = \sup_{\pi}\sup_{1 \leq j \leq k}\vert \sum_{i=1}^j(f_{exp,\pi(i)}-f_{obs,\pi(i)})\vert$$ where $\pi$ is a permutation of the order of the categories, $f_{.,i}$ are the observed and expected frequencies (or equivalently, proportion of observations) in category $i$. This can be written equivalently as: $$D_n = \frac{1}{2} \sum_{i=1}^k\vert f_{exp,i}-f_{obs,i} \vert$$ I would like to extend this to a two-sample case using a randomising/permutation procedure, such: $$D_n^{(r)} = \frac{1}{2} \sum_{i=1}^k\vert f^{(r)}_{\text{group1},i}-f^{(r)}_{\text{group2},i} \vert,\, r=1,\dots,R$$ where $.^{(r)}$ denotes a statistic calculated based on the $r^{\text{th}}$ permutation of the categorical variable. Reject if the value of the original statistic is larger than the value of $95\%$ of the permuted statistics.

Any comments as to the pros/cons/validity of such a procedure are very welcome. Thanks.

For example, suppose your categories are integer ranges $[i, i+1)$ indexed by $i$ and you are observing normal variates of unit variance but unknown mean. 100 observations of a standard normal variate, say, will mainly occupy categories $-2$ through $1$, although you can expect a few to occupy categories $-3$ and $2$. Even for a whopping big shift of $5$ standard errors (i.e., a change in mean of $5/\sqrt{100} = 0.5$), the power of your K-D-like test is only about 50% (when $\alpha = 0.05$).
• if i understand correctly what you wrote, wouldn't $D_n^{(r)}$ be the same for all $r$? also - i can see how to get a monte-carlo estimated critical value for $D_n$; but how about for $D_n^{(r)}$? Dec 21, 2010 at 3:36
• @ronaf Could you provide more detail about $D_n^{(r)}$? What is R? I don't see that permuting the categories does anything at all: notice that no permutation will change the sum of absolute differences of their counts.