Up to what sample size does Anderson-Darling Test gives reliable results on p-value?

As well as I have come across this statement for Anderson-Darling Test:

Small samples sizes tend to “fail to reject” just as *very large* sample sizes tend to reject the null hypothesis. Is it correct?

I have run the normality test on this very large sample size of 6362620. Are the p values reliable for me to make conclusions or is it because the sample size is too large, the p values become unreliable.

The results obtained:

enter image description here

  • 5
    $\begingroup$ 6 million points, almost certainly "too much data" for any test you can think of. You will have to interpret the results yourself on this one. $\endgroup$ Jan 6 at 13:59
  • 10
    $\begingroup$ It's not that the "p-values become unreliable" - quite the opposite, in fact. Rather, it's that the differences that the test is able to detect become so small that they are not practically relevant. $\endgroup$
    – Limey
    Jan 6 at 14:06
  • 8
    $\begingroup$ In essence, significance tests were invented because researchers are human and highly likely to over-interpret evidence from small samples that could just be a side-effect of random variation. They are essentially useless for large samples. It's a normal quantile plot that is almost the only useful evidence here. (Skewness and kurtosis might help a bit.) Besides, none of these tests can cope with whatever other structure is present in your data, e.g. serial or cluster dependence. $\endgroup$
    – Nick Cox
    Jan 6 at 14:14
  • 2
    $\begingroup$ Vote to reope, I don't think linked question answers the direct question which admittedly has very similar flavors. $\endgroup$
    – AdamO
    Jan 9 at 17:21
  • 2
    $\begingroup$ What would be the point of running the test? $\endgroup$
    – whuber
    Jan 9 at 19:04

1 Answer 1


In general, there is no theoretical upper bound on tests. This is a fundamental property of a good statistical test called consistency, i.e. a test is consistent if the power goes to 1 as n goes to infinity.

Most statisticians (myself included) rarely encounter the "problem" of having "n too big", and when presented with the problem, they're quick to say a standard approach would be stupid but not why. If we consider for a moment there are quite a few applications where this might happen. Suppose for instance, I am an industrial statistician measuring the clearance of interference valves on an engine to estimate the risk of the valve hitting the piston, I can design an experiment to run the engine at 12,000 RPMs for 1 hour using high speed cameras and get precision estimates for the clearances. The distribution will give me a probability of wear or give in the components causing a catastrophic failure.

To sum up I would propose 2 approaches.

  1. Put your alpha insanely low. This is a particularly good approach when using global tests such as AD. The point is that it's useless to declare a distribution as non-normal if it's practically very normal. That "choice" is very hard. You might just need to explore some simulations at various sample sizes, develop some metrics, and defend some subjective arguments. You can even say, "alpha=0.05 would work for N=100, so for N=100,000 I will choose alpha = 0.00005". The p value is not unstable, although your software needs some help reporting the actual p-value - use ODS export to get its result, or set alpha in the / options section.
  2. Use estimation instead of inference. With so many $n$, you can just get a kernel smoothed estimate of the density and say, "There it is!" without needing to declare whether it's normal or any other distribution. All the univariate statistics, like the mean, SD, IQR, min, max, can be viewed as probabilistic summaries of the empirical distribution function, so the idea of taking a sample and defining a distribution around is well defined.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.