Finding the difference between a normally distributed random number and randn with an offset using Kolmogorov-Smirnov test and Chi-square test

A normally disturbed random number is generated using randn function in Python. The KS test and Chi-square test is performed to check if the generated numbers belong to a Gaussian (zero mean, unit variance) distribution based on p-value. Then, an offset of 0.01 is added to the generated random numbers and randomness test is performed. The idea is to find the difference between the correct randn and the offset ones using the KS and Chi^2 test.

Here is the code for the KS test. The bin size is fixed (10) for various size of random numbers and the offset is 0.01:

def rand_norm(n=1):
x = nrand.normal(0.0,1,n);
y = 0.01*ones(n) + x;
return x,y

n = arange(100,100001,1000);

a = zeros(len(n));
b = zeros(len(n));
c = zeros(len(n));
d = zeros(len(n));
bin_value = 10;

for i in range(len(n)):
x,y = rand_norm(n[i]);

hist1, bin_edge1 =histogram(x,bins = bin_value, density = 1);
hist2, bin_edge2 =histogram(y,bins = bin_value, density = 1);

a[i],b[i]=stats.kstest(hist1,'norm',N = len(hist1));
c[i],d[i]=stats.kstest(hist2,'norm',N = len(hist2));

figure();
semilogy(n,b)
xlabel('Number of random variables')
ylabel('p-value')
title('KS Test for correct randn');

figure();
semilogy(n,d)
xlabel('Number of random variables')
ylabel('p-value')
title('KS Test for offset randn');


The p value of the KS test for both the cases are plotted in semi-log scale.

There is no difference between the p-values for both the cases, verified by plotting them together. I repeated the same experiment with chi-square function in python. Even Chi-square is giving same p-values for both the case.

Not sure where I am going wrong. It would be a great if someone can help! Thanks in advance!

1 Answer

You do not explain your question in English, as much as in Python, of which I am not a native speaker. (It is possible you have a mistake in your Python code, and I'm not the person to help with that.) So perhaps I'm missing your main point.

However, it seems to me you may be asking too much of the K-S test and chi-squared tests in this setting. (For example, the K-S test is pretty good at detecting differences in shape, but not so good at detecting shift.)

Distinguishing between two normal samples, with a shift in one mean of only $$\Delta=0.01,$$ will take very large sample sizes and a test with potentially good power. Neither the K-S test nor a chi-squared test with binned data has outstanding power.

I believe a two-sample pooled t test would be ideal for detecting a small shift, when we know population variances are identical. Even so it will require a huge sample size reliably to detect a difference of $$0.01.$$

Here is a simulation in R of the power (about 88%) of the pooled t test to detect the larger difference $$\Delta=0.1$$ using samples of size $$n=2000.$$

set.seed(2020)
n = 2000
pv = replicate(10^5, t.test(rnorm(n),rnorm(n)+.1,var.eq=T)$p.val) mean(pv <= .05) [1] 0.88357  By contrast, here is a parallel simulation for the K-S test: set.seed(2020) n = 2000 pv = replicate(10^5, ks.test(rnorm(n),rnorm(n)+.1)$p.val)
mean(pv <= .05)
[1] 0.77775


Without the $$\Delta = 0.1$$ offset, both simulations approximate the probability of rejection to be about $$0.05,$$ as is appropriate.

Note: The test statistic $$D$$ of the 2-sample K-S test is the vertical difference between ECDFs of the two samples: here, samples of size $$50$$ from $$\mathsf{Norm}(0, 1)$$ and $$\mathsf{Norm}(0.1, 1).$$ For such small sample sizes, it takes $$D > 0.22$$ to reject. (The vertical jumps in the ECDFs alone is 0.02.)

set.seed(710)
x = rnorm(50);  y = rnorm(50,.1,1)
ks.test(x,y)

Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.22, p-value = 0.1786
alternative hypothesis: two-sided

plot(ecdf(x), col="blue",
main="ECDFs of NORM(0,1) and NORM(.1,1)")
lines(ecdf(rnorm(y)), col="orange")