A normally disturbed random number is generated using randn function in Python. The KS test and Chi-square test is performed to check if the generated numbers belong to a Gaussian (zero mean, unit variance) distribution based on p-value. Then, an offset of 0.01 is added to the generated random numbers and randomness test is performed. The idea is to find the difference between the correct randn and the offset ones using the KS and Chi^2 test.

Here is the code for the KS test. The bin size is fixed (10) for various size of random numbers and the offset is 0.01:

def rand_norm(n=1):
    x = nrand.normal(0.0,1,n);
    y = 0.01*ones(n) + x;
    return x,y
n = arange(100,100001,1000);

a = zeros(len(n));
b = zeros(len(n));
c = zeros(len(n));
d = zeros(len(n));
bin_value = 10;

for i in range(len(n)): 
    x,y = rand_norm(n[i]);
    hist1, bin_edge1 =histogram(x,bins = bin_value, density = 1);
    hist2, bin_edge2 =histogram(y,bins = bin_value, density = 1);

    a[i],b[i]=stats.kstest(hist1,'norm',N = len(hist1));
    c[i],d[i]=stats.kstest(hist2,'norm',N = len(hist2));

xlabel('Number of random variables')
title('KS Test for correct randn');

xlabel('Number of random variables')
title('KS Test for offset randn');

The p value of the KS test for both the cases are plotted in semi-log scale.

There is no difference between the p-values for both the cases, verified by plotting them together. I repeated the same experiment with chi-square function in python. Even Chi-square is giving same p-values for both the case.

Not sure where I am going wrong. It would be a great if someone can help! Thanks in advance!


1 Answer 1


You do not explain your question in English, as much as in Python, of which I am not a native speaker. (It is possible you have a mistake in your Python code, and I'm not the person to help with that.) So perhaps I'm missing your main point.

However, it seems to me you may be asking too much of the K-S test and chi-squared tests in this setting. (For example, the K-S test is pretty good at detecting differences in shape, but not so good at detecting shift.)

Distinguishing between two normal samples, with a shift in one mean of only $\Delta=0.01,$ will take very large sample sizes and a test with potentially good power. Neither the K-S test nor a chi-squared test with binned data has outstanding power.

I believe a two-sample pooled t test would be ideal for detecting a small shift, when we know population variances are identical. Even so it will require a huge sample size reliably to detect a difference of $0.01.$

Here is a simulation in R of the power (about 88%) of the pooled t test to detect the larger difference $\Delta=0.1$ using samples of size $n=2000.$

n = 2000
pv = replicate(10^5, t.test(rnorm(n),rnorm(n)+.1,var.eq=T)$p.val)
mean(pv <= .05)
[1] 0.88357

By contrast, here is a parallel simulation for the K-S test:

n = 2000
pv = replicate(10^5, ks.test(rnorm(n),rnorm(n)+.1)$p.val)
mean(pv <= .05)
[1] 0.77775

Without the $\Delta = 0.1$ offset, both simulations approximate the probability of rejection to be about $0.05,$ as is appropriate.

Note: The test statistic $D$ of the 2-sample K-S test is the vertical difference between ECDFs of the two samples: here, samples of size $50$ from $\mathsf{Norm}(0, 1)$ and $\mathsf{Norm}(0.1, 1).$ For such small sample sizes, it takes $D > 0.22$ to reject. (The vertical jumps in the ECDFs alone is 0.02.)

x = rnorm(50);  y = rnorm(50,.1,1)

        Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.22, p-value = 0.1786
alternative hypothesis: two-sided

plot(ecdf(x), col="blue", 
     main="ECDFs of NORM(0,1) and NORM(.1,1)")
 lines(ecdf(rnorm(y)), col="orange")

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.