Check if data (N datapoints) originate from known distribution

Assuming that we have N data points $$x_1, \ldots, x_N$$ and a known distribution $$F_0(x)$$ (not a Normal Distribution), i.e. we want to check if the system under study behaves 'typically'.

I would like to quantify the probability that those $$N$$ datapoints originate from this distribution. For large sample sizes, there are various statistical hypothesis tests that could be applied (e.g. chi-squared). However, if the sample size is small, the chi-square test will fail.

Intuitively, it should be possible to detect if those data points are clearly deviating, even with a small sample size. However, is there a mathematical test designed for these conditions?

• "Intuitively, it should be possible to detect if those data points are clearly deviating, even with a small sample size." Identifying the population from which a small sample was taken can be problematic. – BruceET Jul 22 at 17:30

Maybe Kolmogorov-Smirnov test with correction for small samples provided by Jan Vrbik in: Vrbik, Jan (2018). "Small-Sample Corrections to Kolmogorov–Smirnov Test Statistic". Pioneer Journal of Theoretical and Applied Statistics. 15 (1–2): 15–23.

Correction itself is also described on Wikipedia site for Kolmogorov-Smirnov test: replace $$D_N$$ with

$$D_N+{\frac {1}{6{\sqrt {N}}}}+{\frac {D_N-1}{4N}}$$

where $$D_N$$ is standard Kolmogorov-Smirnov statistic.

Use R to generate 10 observations from a standard uniform distribution:

set seed(722)  # for reproducibility
x = runif(10)
summary(x); sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.1270  0.4940  0.7454  0.6627  0.9070  0.9477
 0.293335   # SD

Use the Kolmogorov-Smirnov test to see if the sample is consistent with standard uniform. Appropriately, the answer is Yes because data were sampled from a standard uniform distribution: Large P-value, no rejection.

ks.test(x, punif)

One-sample Kolmogorov-Smirnov test

data:  x
D = 0.31507, p-value = 0.2217
alternative hypothesis: two-sided

Is the sample also consistent with $$\mathsf{Norm}(.5, \sqrt{1/12})?$$ The mean and variance match, but shapes differ. Notice that the parameters mean and standard deviation are specified. Again consistent, but we know the normal distribution is not correct.

ks.test(x, pnorm, .5, sqrt(1/12))

One-sample Kolmogorov-Smirnov test

data:  x
D = 0.36246, p-value = 0.1104
alternative hypothesis: two-sided

However, the K-S test easily rejects that this sample is from $$\mathsf{Exp}(rate=2),$$ which has mean $$1/2 = 0.5,$$ but the wrong SD. This exponential distribution has almost 14% of its probability above $$1,$$ but our sample has no observation above 0.948.

ks.test(x, dexp, 2)

One-sample Kolmogorov-Smirnov test

data:  x
D = 1.5513, p-value < 2.2e-16
alternative hypothesis: two-sided

Notes: (1) See other pages on this site and the Internet, including the relevant Wikipedia page, which has a brief explanation of the test and some remarks about cases in which parameters must be estimated from data.

(2) Several well-known statistical software programs have procedures that check a sample against a list of often used distributions to estimate parameters and see if any distribution is a fit. Often these are called 'distribution ID' procedures and sometimes they are restricted to non-negative data.

For example, when the distribution ID procedure in Minitab is asked to compare the small sample above to normal, lognormal, Weibull, and gamma families, here are the parameter estimates:

ML Estimates of Distribution Parameters

Distribution  Location    Shape    Scale
Normal*        0.66265           0.29334
Lognormal*    -0.55937           0.66158
Weibull                 2.62094  0.74268
Gamma                   3.53947  0.18722

* Scale: Adjusted ML estimate

And here are appropriate probability plots with P-values of Anderson-Darling goodness-of-fit tests in legends. The data are clearly inconsistent with distributions in the lognormal family. (2) For very large sample sizes, Kolmogorov-Smirnov, Anderson-Darling and other goodness-of-fit tests can reject some distributions as not fitting---even when the fit might be good enough for some practical applications.