The Anderson-Darling test statistic is $$A^2 := n \int_{-\infty}^{\infty} \frac{(F_n(x)-F(x))^2}{F(x)(1-F(x))} \, \mathrm{d}F(x)$$ We know that the samples $$Z(x) := \frac{\sqrt{n}(F_{n}(x) - F(x))}{F(x)(1-F(x))} \to \mathcal{N}(0,1)$$ and hence if we convert the expression for $$A^2$$ into a quadrature sum we obtain $$A^2 \approx \sum Z(x_i)^2 F'(x_i) w_i$$ Presumably $$F'(x)$$ decays in some way (say, as a bell curve), so that the the following argument is not totally rigorous, but in any case, we make the approximation that $$A^2 \approx C\sum Z(x_i)^2$$ we see that the statistic is the sum of squares of $$\mathcal{N}(0,1)$$ samples and hence is itself highly variable. Assuming roughly $$k$$ samples of the quadrature sum are located near the maximum of the pdf, then $$A^2 \sim \chi(k)$$, and $$A^2$$ has variance $$k$$, but damped a bit by the shape of the pdf. Even if this argument is not rigorous, we can see that the test is itself is highly variable in this simulation, created with the following Mathematica command:

ListPlot[Table[
AndersonDarlingTest[RandomVariate[NormalDistribution[], n],
Automatic, "TestStatistic"], {n, 2, 10000, 1}]] Moreover, it doesn't improve with n. You have the expectation that if $$F_{n} \to G \ne F$$, then $$A^2$$ should approach some finite positive limit, and if $$F_n \to F$$, then $$A^2 \to 0$$, and hence you're convinced that the distance measure induced by the integral shows that they are the same. But that's not the case! If $$F_n\to G$$, then the integral diverges, and $$F_n \to F$$, the integral bounces around. And maybe you say what's the problem? I can distinguish between a bounded random sequence and an unbounded one. But that's only if you have the ability to arbitrarily draw more and more data. That's not the case in many circumstances.

However, $$A^2/n \to 0$$ iff $$F_n \to F$$, and $$A^2/n \to \epsilon > 0$$ whenever $$F_n \to G \ne F$$. So why don't we consider the quantity $$A^2/n$$ instead of $$A^2$$ in the Anderson-Darling test?

Edit: The conversion to the $$p$$-value seems especially dubious here. You'll get a different p-value every time if the hypothesis is true, and that $$p$$-value looks uniformly distributed on $$[0,1]$$ independent of $$n$$. Fortunately if it's false and $$n \to \infty$$ the $$p$$-value does seem to drop to zero.

Here's the code:

ListPlot[{Table[
AndersonDarlingTest[RandomVariate[NormalDistribution[0.0, 1], n],
NormalDistribution[0, 1], "PValue"], {n, 2, 1000, 1}],
Table[AndersonDarlingTest[
RandomVariate[NormalDistribution[0.2, 1], n],
NormalDistribution[0, 1], "PValue"], {n, 2, 1000, 1}]}] • This argument seems to assume independence among the terms in sum. F_n(x_i) and F_n(X_j) are dependent. Also the simulation you provided was over a the null hypothesis. It will in that case, converge to a distribution with quantiles 0.576 (85%) 0.656 (90%) 0.787 (95%) 0.918 (97.5%) 1.092 (99%)
– Josh
Sep 27 '19 at 17:01
• You should get a new p-value every time under the null. The p-values should be uniformly distributed under the null.
– Josh
Sep 27 '19 at 19:53