Is it reasonable to use the Kolmogorov-Smirnov test to assess the normality of a random variable?

Question

Context

I am confused by the following post where the accepted answer states that :

You can't really even compare the two since the Kolmogorov-Smirnov is for a completely specified distribution (so if you're testing normality, you must specify the mean and variance; they can't be estimated from the data*), while the Shapiro-Wilk is for normality, with unspecified mean and variance.

• you also can't standardize by using estimated parameters and test for standard normal; that's actually the same thing.

Question

Imagine that I have a random sample of measurements X which I standardise using its sample mean and variance. May I use the Kolmogorov Smirnov test as a GOF test to assess normality of this random sample ?

$$ H_0 : X_{scaled} \sim N(0,1) $$

Illustration

To illustrate my question here is a code snippet in R :

# We wish to do a Goodness of Fit test that X is a random sample from a Normal Distribution N(mu,sigma^2)
X <- c(10.212, 10.103, 10.242, 10.106, 10.102, 10.095, 10.042, 10.093, 10.302, 10.111)
sample.mean <- mean(X)
sample.variance <- var(X)
# Or that standardized X (scaled.X) is a random sample from a standard normal distribution N(0,1)
scaled.X <- (X-sample.mean)/(sqrt(sample.variance))


# Kolmogorov-Smirnov Test H0 : X ~ N(0,1)

ks.test(scaled.X,alternative="two.sided",y = "pnorm")
# Do not reject the null.
# Shapiro Test 
shapiro.test(scaled.X)
# Do reject the null.

Note that the KS test and the Shapiro-Wilk test gives contradictory results, hinting towards the Shapiro Wilk test being more powerful in this specific case. This is however not my main question although any comments on this is gladly welcomed.

The specific area of interest of this question is if using the KS test on a standardized random sample (with sample statistics) a sound way to evaluate the normality assumption.

Answer 1

Your approach is Procrustean: when you standardized the data, you forced them to look a little more like standard Normal values than they had. After all, part of detecting a difference in distribution involves comparing their means and variances, which you have forced to be the same.

As a result, you are fooling the KS test. It turns out the p-values it returns are dramatically too large, as these results of 10,000 simulated datasets (of size $50$) attest. They summarize two p-values: one obtained by applying the KS test to an iid standard Normal sample and another obtained in exactly the same way, after standardizing that sample.

The red lines plot the ideal null (uniform) distribution for reference.

One thought would be to correct the standardized p-value somehow. But sometimes the p-values are nearly the same because the original sample happened to be nearly standardized, anyway. On rare occasions the standardization makes the data look less like they were drawn from a standard Normal distribution: the KS test evaluates many other aspects of the distribution than its first two moments. But most often, standardization pulls the p-value up (making it harder to detect a departure from being standard Normal). Consequently, we cannot even predict the correct p-value from the incorrect one with acceptable accuracy. Here is the scatterplot of the pairs of p-values in the simulation.

These considerations are sufficiently general--they appeal to no particular property of the KS test apart from its purpose--and thereby suggest similar problems would attend the use of standardization with almost any distributional test.

---

Such simulations take little time (this requires less than a second to complete) and can be coded in minutes, so they often are worth doing when subtle questions of this kind arise. As an example of how little effort might be needed, here's R code to reproduce this simulation.

n.sim <- 1e4
n <- 50
set.seed(17)
X <- matrix(rnorm(n*n.sim), n)

f <- function(x) ks.test(x, "pnorm")$p.value
ks.1 <- apply(X, 2, f)
ks.2 <- apply(scale(X), 2, f)

The rest of it is a matter of post-processing the arrays of p-values in ks.1 and ks.2. For the record, here's how I did that to make the figures.

# Figure 1: Histograms
par(mfrow=c(1,2))
b <- seq(0, 1, by=0.05)
hist(ks.1, breaks=b, freq=FALSE, col=gray(.9), main="Non-standardized", xlab="p-value")
abline(h=1, lwd=2, col=hsv(0,1,3/4))
hist(ks.2, breaks=b, freq=FALSE, col=gray(.9), main="Standardized", xlab="p-value")
abline(h=1, lwd=2, col=hsv(0,1,3/4))
par(mfrow=c(1,1))

# Figure 2: Scatterplot
plot(ks.1, ks.2, pch=21, bg=gray(0, alpha=.05), col=gray(0, alpha=.2), cex=.5,
     xlab="Non-standardized p-value", ylab="Standardized p-value", asp=1)

Answer 2

Consider the following simulation, in which the K-S test is asked whether a sample of size $n=20$ from $\mathsf{Beta}(2,2)$ is normal. Bogus variants with sample mean and variance and (similarly) standardizing with sample mean and variance are also tried. [Approximate variants of the K-S test using sample mean and SD are available.]

Finally, the Shapiro-Wilk test is asked whether the beta data fit any normal distribution. (Poor power with small $n.)$

set.seed(2022)
m = 10^5; pv1 = pv2 = pv3 = pv4 = numeric(m)
for (i in 1:m) {
 x = rbeta(50, 2,2)
 pv1[i] = ks.test(x, pbeta, 2,2)$p.val
 pv2[i] = ks.test(x, pnorm, mean(x), sd(x))$p.val
 z = (x -mean(x))/sd(x)
 pv3[i] = ks.test(z, pnorm)$p.val
 pv4[i] = shapiro.test(x)$p.val
}
mean(pv1 <=.05) # aprx K-S P-val for BETA(2,2)
[1] 0.05052       # power aprx same as sig level
mean(pv2 <=.05) # dishonest K-S, for NORM(.5,.2236)
[1] 0.00012       # power below sig. level
mean(pv3 <=.05) # dishonest K-S, for standardized bets
[1] 0.00012       # no power, same as above
mean(pv4 <= 10) # honest Shapiro P-val
[1] 0.15131       # aprx power 15%

The two bogus attempts to trick the K-S test (pv2 and pv3) are dysfunctional, giving P-values below the nominal significance level of 5%. [As @whuber remarks, the 'standardization' makes the sample look 'too much' like standard normal.]

About 15% of the time, the Shapiro-Wilk test correctly recognizes that the beta data [all in $(0,1)]$ are inconsistent with any normal distribution.

Reference: The idea to use $\mathsf{Beta}(2,2)$ in my simulation comes from Figure 1(a) of this relevant paper.

Answer 3

In general, statistical tests consist of two parts: calculating a statistic, and then finding a p-value based on that statistic. You certainly can calculate the KS statistic for a normalized sample, but where you run into trouble is calculating the p-value. The published p-value tables for KS are for running a sample against a pre-determined distribution. By standardizing the sample, you're doing something akin to p-hacking, in that you're essentially altering your null hypothesis after seeing your data (the values of the mean and standard deviation for your proposed distribution are part of the null hypothesis, and you're calculating those from the sample).

I find whuber's characterization "when you standardized the data, you forced them to look a little more like standard Normal values than they had" to be possibly misleading, at least if the reader misses the "standard" part. The Shapiro-Wilk test looks at how well the sample matches any normal distribution, not just the standard normal distribution, and standardizing the data doesn't change the Shapiro-Wilk statistic. The Kolmogorov-Smirnov test, on the other hand, is comparing the sample to a particular distribution, not just to the entire class of normal distributions, and so standardization, by forcing the sample to match a particular mean and standard deviation (namely, $0$ and $1$ respectively), increases the probability of appearing to match the distribution.

So if you want to do a statistical test using KS on a standardized sample, you'll have to find some source for p-values other than the standard formula. I find whuber's conclusion "we cannot even predict the correct p-value from the incorrect one with acceptable accuracy" to not be supported by the argument that precedes it in their answer, however (at least without reading more into the argument). The mere fact that using a different distribution for p-values sometimes increases the p-value and sometimes decreases it does not preclude the use of that different distribution (although there may be additional issues in this particular case that make it impractical). If your statistical test has a particular $\alpha$, then any test that has a rejection region with probability mass less than or equal to $\alpha$ is valid, albeit not necessarily useful.

It is likely, however, that to calculate a p-value, you would need some metadistribution, where the actual distribution you're sampling from is modeled as being randomly chosen from some space of distributions, with some meta-probability distribution over that space. There may be some upper bound for the p-values regardless of the metadistribution, however, in which case you can use that. I'm not familiar with what work, if any, has been done establishing p-values for this case. And given that we already have the Shapiro-Wilk test, you might as well use that.