7
$\begingroup$

Is there an equivalent of the two-sample Kolmogorov-Smirnov test for integer data (not count data, as it can include negative integers)?

The Kolmogorov-Smirnov test does not perform well in the presence of lots of ties, which are obviously common with integers.

Improve this question
$\endgroup$
2

2 Answers 2

Reset to default
9
$\begingroup$

The Permutation test could be applied here as well. The idea is as follows.

Let $X_1,...,X_m\sim F$ and $Y_1,...,Y_n\sim G$ be two independent samples and consider testing the hypothesis $H_0:F=G$ vs. $H_1:F\neq G$. For this purpose, label your data as follows

\begin{array}{c c} 1 & X_1\\ 1 & X_2\\ \vdots & \vdots\\ 1 & X_m\\ 2 & Y_1\\ 2 & Y_2\\ \vdots & \vdots\\ 2 & Y_n\\ \end{array}

Now, let $T$ be an statistic of the sample $S=\{X_1,...,X_m,Y_1,...,Y_n\}$ and the labels $L=\{1,1,...,2,2,...,2\}$.

If $H_0$ is true, then the labeling is superfluous.

Now, permute the group labels and recalculate the test statistic a large number of times, say $B$.

The one-sided p-value of this test is calculated as the proportion of sampled permutations where the difference in means was greater than or equal to $T(S,L)$. The two-sided p-value of the test is calculated as the proportion of sampled permutations where the absolute difference was greater than or equal to $\mbox{abs}(T(S,L))$. See

A toy example

Let $X_i \sim \text{Poisson}(10)$, $i=1,...,m=100$, and $Y_j \sim \text{Poisson}(11)$, $j=1,...,n=100$. Consider the statistic $T=\text{mean of Group 1} - \text{mean of Group 2}$. The permutation method using this statistic is implemented below.

rm(list=ls)
set.seed(1)
# Sample size
ns=100
#Simulated data
x = rpois(ns,11)
y = rpois(ns,10)

# Observed statistic    
T.obs = mean(x) - mean(y)

# Pooled data
SL = rbind(cbind(rep(1,ns),x),cbind(rep(2,ns),y))

# Resampling
B=10000
T = rep(0,B)

for(i in 1:B){
samp = sample(SL[,1])
ind1 = which(samp==1)
ind2 = which(samp==2)
T[i] = mean( SL[ind1,2] )- mean( SL[ind2,2] )
}

# p-value
p.value = length(which(abs(T)>abs(T.obs)))/B

I do not know how robust is this method, but after some experiments it seems to perform moderately well. Note that the choice of the statitic $T$ is open and therefore one must be careful on making a meaningful choice in the context of your problem as the performance depends on both the statistic and the sample size.

I hope this helps.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ +1 The usual chi-squared statistic works well. I examined the distributions of bootstrapped p-values for zero-mean shifted Poisson distributions ($X + \lfloor\lambda\rfloor \sim \text{Poisson}(\lambda)$) and found good power even with moderately small sample sizes. E.g., with two datasets of $100$ values each, $\lambda=1$ is discriminated from $\lambda=1.4$ with 50% power at $\alpha=.05$. These chi-squared statistics do not appear to have chi-squared distributions, whence the need to bootstrap the p-values. $\endgroup$
    – whuber ♦
    Commented Sep 7, 2012 at 15:56
1
$\begingroup$

I would suggest the two sample chi square test where you bin the data and compare the binned total with an "expected number" that would fall within the binbased on the pooled sample. This has a generalization to k greater than 2. I am assuming that you are not requiring another test of the emprical cdf form. I think that entire class of test could have some trouble when there are a lot of ties.

Here is a reference that shows you precisely how the two-sample chi square test statistic is calculated along with the degrees of freedom for the asymptotic chi square distirbution.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Do you suggest that each integer is a bin, or to group integers into bins? If its the latter, is there any rule of thumb for selecting an appropriate number of bins? $\endgroup$
    – fmark
    Commented Sep 7, 2012 at 0:43
  • $\begingroup$ @fmark Of course a bin would include a group of integers. There is no rule of thumb that I can think of that would work in general. The idea is to have enough bins so that the histogram is not too smooth and not overly variable. It is very much like choosing a bandwidth for a kernel smoother. $\endgroup$
    – Michael R. Chernick
    Commented Sep 7, 2012 at 0:48
  • 2
    $\begingroup$ How does one apply the chi-square test for a two-sample application? You need a reference distribution but you don't have one in that case. How do you find it? What would be the degrees of freedom to use? $\endgroup$
    – whuber ♦
    Commented Sep 7, 2012 at 14:56
  • 2
    $\begingroup$ I did some experimentation and found the distribution doesn't look chi-square even for some largish datasets (e.g., comparing $1000$ values to $100$ values and each bin with more than $5$ values (typically).) $\endgroup$
    – whuber ♦
    Commented Sep 7, 2012 at 15:33
  • 2
    $\begingroup$ Are you suggesting I didn't know this? :-) But because in practice we tend to get small counts out in the tails, the chi square approximation doesn't work well. A permutation test (or, simply, bootstrapping the chi square statistic) does work: that's what I did the experimentation to confirm. $\endgroup$
    – whuber ♦
    Commented Sep 7, 2012 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.