While it is easier to use the Pearson chi-square/Cressie-Read type test, I would like to test the equality of proportions in $k$ categories across two groups using a Kolmogorov-Smirnov type test of the form proposed by Pettitt & Stephens (1977) (see also here).

In particular as the authors of that paper point out, it may have some power against trending alternatives. So their one-sample nominal/categorical Kolmogorov-Smirnov test has the form: $$ D_n = \sup_{\pi}\sup_{1 \leq j \leq k}\vert \sum_{i=1}^j(f_{exp,\pi(i)}-f_{obs,\pi(i)})\vert$$ where $\pi$ is a permutation of the order of the categories, $f_{.,i}$ are the observed and expected frequencies (or equivalently, proportion of observations) in category $i$. This can be written equivalently as: $$ D_n = \frac{1}{2} \sum_{i=1}^k\vert f_{exp,i}-f_{obs,i} \vert$$ I would like to extend this to a two-sample case using a randomising/permutation procedure, such: $$ D_n^{(r)} = \frac{1}{2} \sum_{i=1}^k\vert f^{(r)}_{\text{group1},i}-f^{(r)}_{\text{group2},i} \vert,\, r=1,\dots,R $$ where $.^{(r)}$ denotes a statistic calculated based on the $r^{\text{th}}$ permutation of the categorical variable. Reject if the value of the original statistic is larger than the value of $95\%$ of the permuted statistics.

Any comments as to the pros/cons/validity of such a procedure are very welcome. Thanks.


1 Answer 1


The answer depends on the nature of the data generation process and on the alternative hypothesis you have in mind.

Your test is a kind of unweighted chi-square. Because of this lack of weighting, changes that principally affect the less-populated categories will be difficult to detect. For example, your test is going to be much less powerful than the chi-square test for a uniform shift in location, which is detected primarily by noticing that almost all the probability in one tail gets shifted into the other tail.

For example, suppose your categories are integer ranges $[i, i+1)$ indexed by $i$ and you are observing normal variates of unit variance but unknown mean. 100 observations of a standard normal variate, say, will mainly occupy categories $-2$ through $1$, although you can expect a few to occupy categories $-3$ and $2$. Even for a whopping big shift of $5$ standard errors (i.e., a change in mean of $5/\sqrt{100} = 0.5$), the power of your K-D-like test is only about 50% (when $\alpha = 0.05$).

It is difficult to conceive of a setting where this test will be more powerful than the chi-square test. If you think you are in such a situation, perform some simulations to find out what the power is and how it compares to the standard alternative tests.

  • $\begingroup$ if i understand correctly what you wrote, wouldn't $D_n^{(r)}$ be the same for all $r$? also - i can see how to get a monte-carlo estimated critical value for $D_n$; but how about for $D_n^{(r)}$? $\endgroup$
    – ronaf
    Dec 21, 2010 at 3:36
  • $\begingroup$ @ronaf Could you provide more detail about $D_n^{(r)}$? What is R? I don't see that permuting the categories does anything at all: notice that no permutation will change the sum of absolute differences of their counts. $\endgroup$
    – whuber
    Dec 21, 2010 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.