Suppose I have two sets of random numbers drawn from the Gaussian distribution, $\mathbf{r_1}$ and $\mathbf{r_2}$. I then do a 2-sample Kolmogorov-Smirnov test (KS test) between the two to determine if the two sets are drawn from the same distribution with some confidence level, $\alpha$ (e.g. 0.05).
Now, in reality, the two sets are drawn from the same distribution, so I thought that the KS-test would always return the null hypothesis (that they are drawn from the same distribution).
But what I find is that, if I repeat this procedure again and again, the percentage of times that the KS-test rejects the null hypothesis converges to $\alpha$.
What is also strange is that, if I plot $\mathbf{r_1}$ vs. $\mathbf{r_2}$ for a run which has a rejection and then a second plot of the same but for a run which has no rejection, there is no discernible difference in the two plots:
1) Why does the KS test reject the hypothesis sometimes even if the two are, in reality, drawn from the same distribution?
2) Why does the first plot accept the the null hypothesis while the second plot rejects it? There is no visual difference between the two so it seems very unintuitive as to what the difference between the two sets is.
3) Is there any indication on either of the plots as to why one is a rejection while the other accepts the null hypothesis?
Code sample from MATLAB:
N = 10000; %number of runs to do
alpha = 0.05;
for i = 1:N %Do N runs
r1(:,i) = randn(1000,1); %random vector 1
r2(:,i) = randn(1000,1); %random vector 2
%KS test
[h(i),p(i),KS(i)] = kstest2(r1(:,i),r2(:,i),'Alpha',alpha);
end
percent_rejected = length(find(h==1))/N; %Percent that reject hypothesis that they
%are drawn from the same distribution (approaches alpha as N -> infty)
%Plot some results: These two figures always "look" the same even though
%they have completely opposite P-values
[val, ind] = min(p); %Find minimum P-value (less than alpha is a rejection
figure(1)
plot(r1(:,ind),r2(:,ind),'.k'); hold on; axis equal
title(['Rejects Null: P value = ',num2str(p(ind))])
xlabel('r_1'); ylabel('r_2')
axis([min(r1(:)) max(r1(:)) min(r2(:)) max(r2(:))])
[val, ind] = max(p); %Find maximum P-value (greater than alpha accepts null)
figure(2)
plot(r1(:,ind),r2(:,ind),'.r'); axis equal
title(['Accepts Null: P value = ',num2str(p(ind))])
xlabel('r_1'); ylabel('r_2')
axis([min(r1(:)) max(r1(:)) min(r2(:)) max(r2(:))])