# Kolmogorov-Smirnov instability depending on whether values are small or big

Out of curiosity, I realised that the Kolmogorov-Smirnov normality test returns two very different p-values depending on whether the dataset has small or large numbers. Is this normal and is there a number size limit for this test? From what I saw, the Shapiro-Wilk test was much more stable.

I tried this

ks.test(c(0.5379796,1.1230795,-0.4047321,-0.8150001,0.9706860),"pnorm")

One-sample Kolmogorov-Smirnov test

data:  c(0.5379796, 1.1230795, -0.4047321, -0.8150001, 0.970686)

D = 0.3047, p-value = 0.6454

alternative hypothesis: two-sided


And then I multiplied each value by 100



ks.test(c(53.79796,112.30795,-40.47321,-81.50001,97.06860),"pnorm")

One-sample Kolmogorov-Smirnov test

data:  c(53.79796, 112.30795, -40.47321, -81.50001, 97.06860)

D = 0.6, p-value = 0.03008

alternative hypothesis: two-sided


With the same data, the Shapiro-Wilk test returns a p-value of 0.3999.

• Okay, those numbers look more like you multiplied to 100. :)
– Dave
Commented Mar 6, 2023 at 9:43
• This question is addressed in multiple posts already on site. Commented Mar 7, 2023 at 23:10

The ways you’ve coded it, you’re asking the KS test about a null hypothesis that the distribution is $$N(0,1)$$. In the first set of numbers, that looks plausible. Consequently, the p-value is high. In the second set of numbers, that does not seem to be the case. Numbers like those don’t typically come from a $$N(0,1)$$ distribution. Consequently, the p-value is low.

By multiplying by a factor, you’ve changed the variance. Since the KS test considers all aspects of the distribution, variance included, the test correctly regards the two data sets as different.

The reason that Shapiro-Wilk is more stable is because it evaluates normality. Multiplying by a positive factor does not change the normality, so Shapiro-Wilk will not have the same kind of sensitivity to a variance change that KS has.

• True, there was a typo in the code for the *100. but your answer still holds, thank you ! Commented Mar 6, 2023 at 9:44
• In addition, there are specific warnings about using the K-S test with parameters estimated from the data, which you presumably need here, and general warnings about the relative uselessness of any such tests. Just plot the data on a normal quantile plot! (Many other names: normal probability plot, normal scores plot, etc.) Commented Mar 6, 2023 at 9:54
• See this for elaboration on what @NickCox wrote about normality testing being of questionable usefulness. // Regarding issues with testing with estimated parameters, check out the Lilliefors test.
– Dave
Commented Mar 6, 2023 at 9:57
• @Xav64 Standardizing your data with the sample mean and variance and then testing if the standardized data are $N(0,1)$ is the same as testing with the original data and using the sample mean and variance. You can check this in a simulation where you will get identical p-values. (I assume you mean the usual z-score by standardizing: $x_{new}=\dfrac{x_{original}-\bar x}{s}$.)
– Dave
Commented Mar 6, 2023 at 9:58
• I don’t totally follow what you’re asking, but that Lilliefors test might be what you want. @Xav64
– Dave
Commented Mar 6, 2023 at 10:09

Adding to the existing response, it's worth noting that the two ks.test calls below produce the same output.

x = c(0.5379796,1.1230795,-0.4047321,-0.8150001,0.9706860)
ks.test(x, pnorm)
#>
#>  Exact one-sample Kolmogorov-Smirnov test
#>
#> data:  x
#> D = 0.3047, p-value = 0.6454
#> alternative hypothesis: two-sided

ks.test(x*100, pnorm, sd = 100)
#>
#>  Exact one-sample Kolmogorov-Smirnov test
#>
#> data:  x * 100
#> D = 0.3047, p-value = 0.6454
#> alternative hypothesis: two-sided


R syntax note: the default arguments to pnorm() are mean = 0, sd = 1. Anything after the second argument in ks.test() gets passed as an argument to the pnorm() function in this case.

• so does it mean that the optimal way of running this test is to write ks.test(x, pnorm, mean=mean(sample), sd=sd(sample)) or is there a faster way? Commented Mar 6, 2023 at 12:04
• Reply to comment: 1. No because that test is invalid. See remarks elsewhere with key word=:Lilliefors. 2. Who said optimal=faster? Commented Mar 6, 2023 at 12:40