I want to use Kolmogorov-Smirnov test to check how given clusters of 1D points differs from normal distribution (original question here: How to test which data match model at best).
I am considering a following approach:
FOREACH cluster p = points FROM cluster n = SIZE(p) mu = AVG(p) sigma = SQRT(VARIANCE(p)) tmp = GENERATE n RANDOM points FROM normal_distribution(mu, sigma) result = KS-TEST(SORT(p), SORT(tmp)) IF result > threshold THEN ok OTHERWISE not ok
I took implementation of KS-TEST from here: http://root.cern.ch/root/html/src/TMath.cxx.html#RDBIQ Number of points is usually hundreds or thousands.
I have observed that result strongly depends on randomly generated "tmp" points. Even when I randomly generated two sets of points from same distribution with same parameters, the resulting probability from KS-TEST floated between 0.0+something and 0.99+something. So it is difficult for me to choose a proper "threshold" value. The same cluster can be once considered as "close-to-normal-distribution" and once not.
Can you give me advice, what am I doing wrong, how can I get more reliable results?