# Equivalence tests for non-normal data?

I have some data that I can't necessarily assume to be drawn from normal distributions, and I would like to conduct tests of equivalence between groups. For normal data, there are techniques like TOST (two one-sided t-tests). Is there anything analogous to TOST for non-normal data?

• I'm not familiar with TOST, but are you looking for Mann-Whitney? This is a nonparametrical test (in the sense that no assumptions on the distributions are made) that may provide evidence that two groups come from different distributions. – Nick Sabbe Mar 21 '13 at 7:52
• I'm looking for a test where the null hypothesis is that there is a difference, and the alternative hypothesis is that there is (almost) no difference. – Ryan C. Thompson Mar 21 '13 at 17:19
• For small samples, you might have a look at the answers in stats.stackexchange.com/questions/49782/…. For larger samples, the classic approach with t tests is fine thanks to the Central Limit Theorem. – Michael M Dec 30 '13 at 10:41
• Nothing in the phrase "Two one-sided tests" - nor the underlying logic implies normal-theory. It should be perfectly possible to adapt it to a location-shift alternative with a non-normal distribution. But beware - in many cases with non-normal data what you really want is a scale-shift kind of equivalence test, and with other kinds of data, something else instead. Knowing what is needed really depends on what you're measuring and what problem you're solving. Rather than trying to squeeze your peg into a round hole, it pays to examine the peg. – Glen_b -Reinstate Monica Jan 29 '14 at 4:53

The logic of TOST employed for Wald-type t and z test statistics (i.e. $$\theta / s_{\theta}$$ and $$\theta / \sigma_{\theta}$$, respectively) can be applied to the z approximations for nonparametric tests like the sign, sign rank, and rank sum tests. For simplicity I assume that equivalence is expressed symmetrically with a single term, but extending my answer to asymmetric equivalence terms is straightforward.

One issue that arises when doing this is that if one is accustomed to expressing the equivalence term (say, $$\Delta$$) in the same units as $$\theta$$, then the the equivalence term must be expressed in units of the particular sign, signed rank, or rank sum statistic, which is both abstruse, and dependent on N.

However, one can also express TOST equivalence terms in units of the test statistic itself. Consider that in TOST, if $$z = \theta/\sigma_{\theta}$$, then $$z_{1} = (\Delta - \theta)/\sigma_{\theta}$$, and $$z_{2} = (\theta + \Delta)/\sigma_{\theta}$$. If we let $$\varepsilon = \Delta / \sigma_{\theta}$$, then $$z_{1} = \varepsilon - z$$, and $$z_{2} = z + \varepsilon$$. (The statistics expressed here are both evaluated in the right tail: $$p_{1} = \text{P}(Z > z_{1})$$ and $$p_{2} = \text{P}(Z > z_{2})$$.) Using units of the z distribution to define the equivalence/relevance threshold may be preferable for non-parametric tests, since the alternative defines the threshold in units of signed-ranks or rank sums, which may be substantively meaningless to researchers and difficult to interpret.

If we recognize that (for symmetric equivalence intervals) it is not possible to reject any TOST null hypothesis when $$\varepsilon \le z_{1-\alpha}$$, then we might proceed to make decisions on appropriate size of the equivalence term accordingly. For example $$\varepsilon = z_{1-\alpha} + 0.5$$.

This approach has been implemented with options for continuity correction, etc. in the package tost for Stata (which now includes specific TOST implementations for the Shapiro-Wilk and Shapiro-Francia tests), which you can access by typing in Stata:

Edit: Why the logic of TOST is sound, and equivalence test formations have been applied to omnibus tests, I have been persuaded that my solution was based on a deep misunderstanding of the approximate statistics for the Shapiro-Wilk and Shapiro-Francia tests

It's not a TOST per se, but the Komolgorov-Smirnov test allows one to test for the significance of the difference between a sample distribution and a second reference distribution you can specify. You can use this test to rule out a specific kind of different distribution, but not different distributions in general (at least, not without controlling for error inflation across tests of all possible alternatives...if that's somehow possible itself). The alternative hypothesis for any one test will remain the less specific "catch-all" hypothesis, as usual.

If you can settle for a test of distributional differences between two groups where the null hypothesis is that the two groups are equivalently distributed, you can use the Komolgorov-Smirnov test to compare one group's distribution to the other group's. That's probably the conventional approach: ignore the differences if they're not statistically significant, and justify this decision with a test statistic.

In any case, you may want to consider some deeper issues arising from the "all-or-nothing" approach to rejecting a null hypothesis. One such issue is very popular here on Cross Validated: "Is normality testing 'essentially useless'?" People like to answer normality-testing questions with a question: "Why do you want to test this?" The intention, I assume, is generally to invalidate the reason for testing, which may ultimately lead in the right direction. The gist of useful responses to the question I've linked here seems to be as follows:

1. If you're concerned about violations of parametric test assumptions, you should just find a nonparametric test that doesn't make distributional assumptions instead. Don't test whether you need to use the nonparametric test; just use it!
2. You should replace the question, "Is my distribution significantly non-normal?" with, "How non-normal is my distribution, and how is this likely to affect my analyses of interest?" For instance, tests regarding central tendency (especially involving means) may be more sensitive to skewness than to kurtosis, and vice versa for tests regarding (co)variance. Nonetheless, there are robust alternatives for most analytic purposes that aren't very sensitive to either kind of non-normality.

If you still wish to pursue a test of equivalence, here's another popular discussion on Cross Validated that involves equivalence testing.

• Equivalence testing is well established, and you misunderstand its null hypotheses, which are generally of the form H$^{-}_{0}: |\theta-\theta_{0}| \ge \Delta$. This is an interval hypothesis which can translate, for example, into two one-sided tests (TOST): H$^{-}_{01}: \theta-\theta_{0} \ge \Delta$, or H$^{-}_{01}: \theta-\theta_{0} \le -\Delta$. If one rejects H$^{-}_{01}$ & H$^{-}_{02}$, then you must conclude that $-\Delta < \theta - \theta_{0} < \Delta$, i.e. that your groups are equivalent within the interval $[-\Delta,\Delta]$. – Alexis Apr 24 '14 at 4:23
• Fair enough; I was probably a bit misleading. I've removed the parts to which you seem to object. However, I think you've worded your comment a bit too strongly. Despite the fact that the forced dichotomous fail to/reject approach is well-established, most samples can't completely preclude the possibility that the null is true. There is almost always some chance of false rejection error if one insists on rejection, which is usually not literally necessary. That was probably the more important point I intended to make originally. Hopefully it's a little clearer now without the deleted stuff – Nick Stauner Apr 24 '14 at 4:41
• Well, in my opinion, the strength of equivalence tests (e.g. H$^{-}_{0}$) comes from combining them with the familiar tests for difference (e.g. H$^{+}_{0}$). Check it out: (1) Reject H$^{+}_{0}$ & Not Reject H$^{-}_{0}$, conclude relevant difference; (2) Not Reject H$^{+}_{0}$ & Reject H$^{-}_{0}$, conclude equivalence (for $\Delta$); (3) Reject H$^{+}_{0}$ & Reject H$^{-}_{0}$, conclude trivial difference (i.e. it's there, but you don't care); and (4) Not Reject H$^{+}_{0}$ & Not Reject H$^{-}_{0}$, conclude indeterminacy_/_underpowered tests. Puts power usefully into the analysis. – Alexis Apr 24 '14 at 4:49
• Of course, issues of sensitivity and specificity, PPV and NPV do not go away. – Alexis Apr 24 '14 at 4:50

Equivalence is never something we can test. Think about the hypothesis: $\mathcal{H}_0: f_x \ne f_y$ vs $\mathcal{H}_1: f_x = f_y$. NHST theory tells us that, under the null, we can choose anything under $\mathcal{H}_0$ that best fits the data. That means we can almost always get arbitrarily close to the distribution. For instance, if I want to test $f_x \sim \mathcal{N}(0, 1)$, the probability model that allows for separate distributions of $\hat{f}_x$ and $\hat{f}_y$ will always be more likely under the null, a violation of critical testing assumptions. Even if the sample $X=Y$ identically, I can get a likelihood ratio that is arbitrarily close to 1 with $f_y \approx f_x$.

If you know a suitable probability model for the data, you can use a penalized information criterion to rank alternate models. One way is to use the BICs of the two probability models (the one estimated under $\mathcal{H}_0$ and $\mathcal{H}_1$. I've used a normal probability model, but you can easily get a BIC from any type of maximum likelihood procedure, either by hand or using the GLM. This Stackoverflow post gets in nitty-gritty for fitting distributions. An example of doing this is here:

set.seed(123)
p <- replicate(1000, { ## generate data under the null
x <- rnorm(100)
g <- sample(0:1, 100, replace=T)
BIC(lm(x~1)) > BIC(lm(x~g))
})
mean(p)


gives

> mean(p)
[1] 0.034


$p$ here is the proportion of times that the BIC of the null model (separate models) is better (lower) than the alternative model (equivalent model). This is remarkably close to the nominal 0.05 level of statistical tests.

On the other hand if we take:

set.seed(123)
p <- replicate(1000, { ## generate data under the null
x <- rnorm(100)
g <- sample(0:1, 100, replace=T)
x <- x + 0.4*g
BIC(lm(x~1)) > BIC(lm(x~g))
})
mean(p)


Gives:

> mean(p)
[1] 0.437


As with NHST there are subtle issues of power and false positive error rates that should be explored with simulation before making definitive conclusions.

I think a similar (perhaps more general method) is using Bayesian stats to compare the posterior estimated under either probability model.

• AdamO you seem to be conflating "testing equality" with "testing equivalence". There is a decades old and solid literature in the methods and application of the latter. – Alexis Apr 3 '18 at 23:24
• See, for example, Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman and Hall/CRC Press, second edition. – Alexis Apr 3 '18 at 23:31
• @Alexis hmm, we don't have access to a library unfortunately. Are you saying equivalence is the same as non-inferiority insofar as estimates lying within a margin are considered equivalent? – AdamO Apr 4 '18 at 14:33
• Not quite: non-inferiority is a one-sided test of whether a new treatment performs no worse than some standard minus a smallest relevant difference that is specified a priori. Tests for equivalence are tests of the null hypothesis that two (or more) quantities are different—in either direction—by more than a smallest relevant difference specified a priori. Some seminal papers: – Alexis Apr 4 '18 at 15:46
• Schuirmann, D. A. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6):657–680. – Alexis Apr 4 '18 at 15:46