Since the null hypothesis is true, the KS test rejects approximately the correct number of times (for any $\alpha$-level, not just $0.05$), as the $U(0,1)$-looking CDF of the p-values shows. I even supercharged the KS test by having a sample size of $25000$, as opposed to your $10000$, yet KS was otnot overpowered.
In addition to the coding mistakes addressed in the other answer, there are two statistics mistakes in the post that I want to address.
If the p-Value is higher than my chosen alpha (5%) my samples are from the distribution.
This is a common misinterpretation of the p-value. We do not accept null hypotheses. When the p-value is larger than $\alpha$, we simply have insufficient evidence to conclude anything. Otherwise, you could just collect two points, conduct your test, pretty much never reject, and keep claiming that you're proving null hypothesis after null hypothesis. Further, this logic applied to all hypothesis testing, not just KS.
I read that the KS-test might not be great for large Data.
There is some truth to this that is discussed extensively in another Cross Validated post. While that question addresses the normal distribution, the logic applies. Summarizing the link, large sample sizes give hypothesis tests (not just KS) great power to detect small differences that are not of practical importance or of interest to clients/customers/reviewers/bosses. However, that only happens when the null hypothesis is slightly incorrect, say a null hypothesis of $\mu = 0$ when the real $\mu = 0.1$. If the null hypothesis is true, the KS test does exactly what it is supposed to, as I will demonstrate in a simulation.
library(ggplot2)
set.seed(2022)
B <- 5000
N <- 25000
ps <- rep(NA, B)
for (i in 1:B){
# Simulate some Weibull data
#
x <- rweibull(N, 2.34, 1)
# KS-test the data for having the specified Weibull distribution
#
ps[i] <- ks.test(x, pweibull, shape = 2.34, scale = 1)$p.value
if (i %% 25 == 0 | i < 5 | B - i < 5){
print(paste(i/B*100, "% complete", sep = ""))
}
}
d <- data.frame(ps = ps, CDF = ecdf(ps)(ps), Distribution = "Weibull")
ggplot(d, aes(x = ps, y = CDF, col = Distribution)) +
geom_line() +
geom_abline(slope = 1, intercept = 0) +
theme_bw()
Since the null hypothesis is true, the KS test rejects approximately the correct number of times (for any $\alpha$-level, not just $0.05$), as the $U(0,1)$-looking CDF of the p-values shows. I even supercharged the KS test by having a sample size of $25000$, as opposed to your $10000$, yet KS was ot overpowered.
Now let's tweak the simulation ever so slightly. A plot above the $y=x$ diagonal line indicates power to detect the difference.
library(ggplot2)
set.seed(2022)
B <- 5000
N <- 25000
ps <- rep(NA, B)
for (i in 1:B){
# Simulate some Weibull data
#
x <- rweibull(N, 2.34, 1)
# KS-test the data for having the specified Weibull distribution
#
ps[i] <- ks.test(x, pweibull, shape = 2.3, scale = 1)$p.value
if (i %% 25 == 0 | i < 5 | B - i < 5){
print(paste(i/B*100, "% complete", sep = ""))
}
}
d <- data.frame(ps = ps, CDF = ecdf(ps)(ps), Distribution = "Weibull 2.3")
ggplot(d, aes(x = ps, y = CDF, col = Distribution)) +
geom_line() +
geom_abline(slope = 1, intercept = 0) +
theme_bw()
I won't tell you if you should care about $2.3$ vs $2.34$, but even if you don't, the KS test sure does!