Tests that Quantify Deviation from Null Hypotheses

Question

I have been delving into non-parametric tests recently, and I've come to realize that most of these tests offer only a partial perspective.

For example, lets say the underlying distribution is $\theta$ and we have a sub-region $R$ in the total space of distributions of concern $\Theta$ (e.g. $R$ being the parameter space of all normal distributions and $\Omega$ being all continuous distributions); lets also say

$$ H_0: \theta \in R $$$$ H_1: \theta \in (\Theta - R) $$

Then most tests out there can only help you reject the null hypothesis $H_0$ when the $p$-value is low (i.e. when if $H_0$ is true, then the probability of obtaining such or more extreme data is low). However, if $H_1$ is true, then many tests say nothing about it.

Let's use a more concrete example. Suppose I want to make sure if $\theta$ represents a normal distribution. So let $R$ be the space of single variate normal distributions, and say I use the Lilliefors test. If $\theta$ is indeed normal, then it is fine. But if $\theta$ is not normal, the test does not necessarily tells me that! Now, imagine I have an investment strategy heavily reliant on the assumption of normality; in such cases, this test becomes almost useless [a].

The tests I'm looking for are akin to the DKW inequality, which is essentially a quantitative version of the Kolmogorov-Smirnov test. Given an $\epsilon > 0$ and an observed sample $S$ of size $n$, it provides you an open neighborhood $N_{\epsilon}(F_S)$ around the empirical cdf $F_{S}$, and tells you that everything out of that neighborhood has likelihood smaller than $2 \exp(-n\epsilon^2)$. In essence, these tests not only identify which distributions align well with the data but also quantify the degree of dissimilarity of others!

Returning to the normality test example: (Q1) Are there any tests available that not only help in rejection but also offer a quantitative measure of how dissimilar the underlying distribution is from normality? Naturally, this quantitative measure is not standardized and should ideally be customizable by the user. Given this need, it's reasonable to inquire about a collection of such tests that can be adapted to various quantitative measures, as long as they are reasonable. Even more generally, I'm hoping for tests for other types of problems, not just normality testing.

The challenge, I suspect, is that developing such tests might be a formidable task, given their complexity. If that's the case, (Q2) is there an established term or category for the type of tests I'm seeking? I'm looking for a more efficient way to locate these tests, but I lack the appropriate terminology.

Footnote

• [a] I understand that there are other normality tests. An example is the Q-Q plot test (problem: not quantitative). For another example, the Shapiro-Wilk test, which is claimed in a 2011 study [1] to have the best power amongst other well-known tests. However, it is still not bullet-proof in the sense that it can't take care of everything in $\Omega$.

Reference

• [1] "Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests" - Razali, Nornadiah; Wah, Yap Bee (2011).

Answer 1

I don't think you want a test at all. A statistical hypothesis test tells you whether you should a) Reject the null or b) Not reject the null. That's all. That's just what tests do.

And the results will depend on how far the results you get are from what would be expected if the null were true, but also on the sample size.

For your specific example:

Now, imagine I have an investment strategy heavily reliant on the assumption of normality; in such cases, this test becomes almost useless

Well, it's hard to imagine any investment strategy that requires exact normality. In addition, it's hard to imagine one that works great if the distribution is very close to normal, but horribly if it is a little bit off.

Rather than a test, I would look at graphs. In particular, I'd look at quantile normal graphs. And, given the type of question you are asking, I'd use both theory and history to try to get a sense of how well the strategy does under different deviations from normal. (I'd hazard a guess that fat tails are dangerous, even before I looked at any data).

There are measures of distance between curves, and those are quantified. There are surely tests based on those distances. But I don't think that's what you want.