# Tests that Quantify Deviation from Null Hypotheses

I have been delving into non-parametric tests recently, and I've come to realize that most of these tests offer only a partial perspective.

For example, lets say the underlying distribution is $$\theta$$ and we have a sub-region $$R$$ in the total space of distributions of concern $$\Theta$$ (e.g. $$R$$ being the parameter space of all normal distributions and $$\Omega$$ being all continuous distributions); lets also say

$$H_0: \theta \in R$$$$H_1: \theta \in (\Theta - R)$$

Then most tests out there can only help you reject the null hypothesis $$H_0$$ when the $$p$$-value is low (i.e. when if $$H_0$$ is true, then the probability of obtaining such or more extreme data is low). However, if $$H_1$$ is true, then many tests say nothing about it.

Let's use a more concrete example. Suppose I want to make sure if $$\theta$$ represents a normal distribution. So let $$R$$ be the space of single variate normal distributions, and say I use the Lilliefors test. If $$\theta$$ is indeed normal, then it is fine. But if $$\theta$$ is not normal, the test does not necessarily tells me that! Now, imagine I have an investment strategy heavily reliant on the assumption of normality; in such cases, this test becomes almost useless [a].

The tests I'm looking for are akin to the DKW inequality, which is essentially a quantitative version of the Kolmogorov-Smirnov test. Given an $$\epsilon > 0$$ and an observed sample $$S$$ of size $$n$$, it provides you an open neighborhood $$N_{\epsilon}(F_S)$$ around the empirical cdf $$F_{S}$$, and tells you that everything out of that neighborhood has likelihood smaller than $$2 \exp(-n\epsilon^2)$$. In essence, these tests not only identify which distributions align well with the data but also quantify the degree of dissimilarity of others!

Returning to the normality test example: (Q1) Are there any tests available that not only help in rejection but also offer a quantitative measure of how dissimilar the underlying distribution is from normality? Naturally, this quantitative measure is not standardized and should ideally be customizable by the user. Given this need, it's reasonable to inquire about a collection of such tests that can be adapted to various quantitative measures, as long as they are reasonable. Even more generally, I'm hoping for tests for other types of problems, not just normality testing.

The challenge, I suspect, is that developing such tests might be a formidable task, given their complexity. If that's the case, (Q2) is there an established term or category for the type of tests I'm seeking? I'm looking for a more efficient way to locate these tests, but I lack the appropriate terminology.

### Footnote

• [a] I understand that there are other normality tests. An example is the Q-Q plot test (problem: not quantitative). For another example, the Shapiro-Wilk test, which is claimed in a 2011 study [1] to have the best power amongst other well-known tests. However, it is still not bullet-proof in the sense that it can't take care of everything in $$\Omega$$.

### Reference

• [1] "Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests" - Razali, Nornadiah; Wah, Yap Bee (2011).
• "Suppose I want to make sure if 𝜃 represents a normal distribution. [...] say I use the Lilliefors test. If 𝜃 is indeed normal, then it is fine. But if 𝜃 is not normal, the test does not necessarily tells me that!": This is exactly what the Lilliefors test would tell you.
– Eoin
Sep 25, 2023 at 15:17
• @Eoin I know, and that's why I used it as a counterexample to illustrate what kind of tests I am looking for. Sep 25, 2023 at 15:27
• I suggest you rewrite your question, in that case, because as written the answers are (1) yes, and you've already listen several of them, and (2) they're called Normality tests. I think part of the problem is that you seem to mean something specific when you say "quantitative", but I don't know what it is.
– Eoin
Sep 25, 2023 at 15:32
• @Eoin Are you suggesting that Lilliefors test is one test that satisfy my question in Q1? I think that's not the case. The Lilliefors test only helps you reject but says nothing about non-normal distributions; whereas in my first question Q1, I asked for "a test [..] [with a] quantitative measure of how dissimilar the underlying distribution is from normality". Sep 25, 2023 at 15:37
• The Kolmogorov-Smirnov test statistic is a "quantitive measure of how dissimilar [the data] is from normality". The Lilliefors test is just a way of figuring how how big the KS test statistic needs to be in order to be significantly different from normality. I don't know what else you could mean by "says nothing about non-normal distributions".
– Eoin
Sep 25, 2023 at 15:53

I don't think you want a test at all. A statistical hypothesis test tells you whether you should a) Reject the null or b) Not reject the null. That's all. That's just what tests do.

And the results will depend on how far the results you get are from what would be expected if the null were true, but also on the sample size.