The KS test does what it is supposed to do, even when you have a million observations. When the null hypothesis is true, the KS test rejects the null about $\alpha$ of the time. What does happen is that, because of the large sample size, there is considerable power to reject slight deviations from the null hypothesis that might not seem to have practical significance.
Thus, use all of your points.
If you do not like that the test has such power because you are rooting for $p>\alpha$, you should think hard about if hypothesis testing is the right tool for your work. Hypothesis testing is extremely literal, and it is a feature, not a bug, that hypothesis testing can detect small deviations from the null hypothesis when the sample size is large. If you want to test if the data are "close" to the null hypothesis, then you might want to think about how to quantify "close" and consider methods like equivalence testing.
You may be interested in the links below.
Is normality testing 'essentially useless'?
Significance test for large sample sizes
EDIT
Let's see a simulation with $100,000$ observations per test (to save computing time).
library(ggplot2)
set.seed(2023)
N <- 1e5
R <- 5000
ps1 <- ps2 <- rep(NA, R)
for (i in 1:R){
# Simulate draws from N(0, 1)
#
x <- rnorm(N, 0, 1)
# KS test if the distribution is N(0, 1) or not, then save the p-value
#
ps1[i] <- ks.test(x, "pnorm", 0, 1)$p.value
# Simulate draws from N(0.01, 1)
#
y <- rnorm(N, 0.01, 1)
# KS test if the distribution is N(0, 1) or not, then save the p-value
#
ps2[i] <- ks.test(y, "pnorm", 0, 1)$p.value
if (i %% 75 == 0 | i < 5 | R - i < 5){
print(paste(
i/R*100,
"% complete",
sep = ""
))
}
}
d1 <- data.frame(
pvalue = c(ps1, ps2),
CDF = ecdf(ps1)(c(ps1, ps2)),
null = "True"
)
d2 <- data.frame(
pvalue = c(ps1, ps2),
CDF = ecdf(ps2)(c(ps1, ps2)),
null = "False"
)
d <- rbind(d1, d2)
ggplot(d, aes(x = pvalue, y = CDF, col= null)) +
geom_point() +
geom_abline(slope = 1, intercept = 0)
When the nul hypothesis is true, despite there being a huge number of observations (you can bump it up to a million and get the same result), the distribution of p-values is $U(0,1)$ (the blue CDF), meaning that there is a probability of $\alpha$ of falsely rejecting a null hypothesis. For instance, when $\alpha = 0.05$, ecdf(ps1)(0.05) shows that the true null hypothesis is rejected $4.8\%$ of the time, just like is supposed to happen.
However, the red graph shows that the KS test indeed has solid power to reject a false null hypothesis, even one that is slightly false $\big(N(0.01, 1)$ is the true distribution, while the null hypothesis is that the distribution is $N(0, 1)\big)$, and ecdf(ps2)(0.05) shows the test to have a power of $77.78\%$ to reject the false null hypothesis.
Importantly, however, the ability to reject this false null hypothesis does not come at the expense of not rejecting a true null hypothesis. The KS test works exactly how a hypothesis test is meant to work.
