0
$\begingroup$

Suppose I have two sets of random numbers drawn from the Gaussian distribution, $\mathbf{r_1}$ and $\mathbf{r_2}$. I then do a 2-sample Kolmogorov-Smirnov test (KS test) between the two to determine if the two sets are drawn from the same distribution with some confidence level, $\alpha$ (e.g. 0.05).

Now, in reality, the two sets are drawn from the same distribution, so I thought that the KS-test would always return the null hypothesis (that they are drawn from the same distribution).

But what I find is that, if I repeat this procedure again and again, the percentage of times that the KS-test rejects the null hypothesis converges to $\alpha$.

What is also strange is that, if I plot $\mathbf{r_1}$ vs. $\mathbf{r_2}$ for a run which has a rejection and then a second plot of the same but for a run which has no rejection, there is no discernible difference in the two plots:

enter image description here

enter image description here

1) Why does the KS test reject the hypothesis sometimes even if the two are, in reality, drawn from the same distribution?

2) Why does the first plot accept the the null hypothesis while the second plot rejects it? There is no visual difference between the two so it seems very unintuitive as to what the difference between the two sets is.

3) Is there any indication on either of the plots as to why one is a rejection while the other accepts the null hypothesis?

Code sample from MATLAB:

N = 10000; %number of runs to do
alpha = 0.05;

for i = 1:N %Do N runs

    r1(:,i) = randn(1000,1); %random vector 1
    r2(:,i) = randn(1000,1); %random vector 2

    %KS test
    [h(i),p(i),KS(i)] = kstest2(r1(:,i),r2(:,i),'Alpha',alpha);

end

percent_rejected = length(find(h==1))/N; %Percent that reject hypothesis that they
    %are drawn from the same distribution (approaches alpha as N -> infty)


%Plot some results: These two figures always "look" the same even though
%they have completely opposite P-values

[val, ind] = min(p); %Find minimum P-value (less than alpha is a rejection
figure(1)
plot(r1(:,ind),r2(:,ind),'.k'); hold on; axis equal
title(['Rejects Null: P value = ',num2str(p(ind))])
xlabel('r_1'); ylabel('r_2')
axis([min(r1(:)) max(r1(:)) min(r2(:)) max(r2(:))])


[val, ind] = max(p); %Find maximum P-value (greater than alpha accepts null)
figure(2)
plot(r1(:,ind),r2(:,ind),'.r'); axis equal
title(['Accepts Null: P value = ',num2str(p(ind))])
xlabel('r_1'); ylabel('r_2')
axis([min(r1(:)) max(r1(:)) min(r2(:)) max(r2(:))])
Improve this question
$\endgroup$
  • $\begingroup$ Displaying scatterplots (which depict bivariate data distributions) in a context that otherwise seems to concern univariate distributions is mysterious and confusing. Could you clarify whether you are comparing univariate distributions or not? If it's the former, what's the purpose of the scatterplots? And if it's the latter, then how have you managed to adapt the KS test to multivariate data? $\endgroup$ – whuber♦ Nov 21 '18 at 20:11
  • 1
    $\begingroup$ @Darcy when you say: "if I repeat this procedure again and again, the percentage of times that the KS-test rejects the null hypothesis converges to $α$" - it should! That's what the significance level is - you are choosing the proportion of times you will reject a true null hypothesis ($\alpha$ is the rate of making a type I error when the null is true). See the second sentence of the wikipedia article on Statistical significance. Rather than surprise, the reaction to that should be relief (that you didn't make a mistake somewhere) $\endgroup$ – Glen_b Nov 21 '18 at 22:36
  • $\begingroup$ @whuber I am doing a 2-sample KS test with one of the samples being $\mathbf{r_1}$ and the other being $\mathbf{r_2}$. I thought the scatter plots were an interesting way of showing the relationship between the two variables and I thought that there would be some noticeable visual difference between scatterplots which rejected the null hypothesis versus scatterplots which did not reject the null. It is possible that my intuition was wrong but I find it strange that there isn't any discernible difference between the plots... $\endgroup$ – Darcy Nov 21 '18 at 22:45
  • $\begingroup$ If anything, the scatterplots obscure any differences between the distributions. Use p-p plots. $\endgroup$ – whuber♦ Nov 22 '18 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.