Equivalence tests for non-normal data?

Question

I have some data that I can't necessarily assume to be drawn from normal distributions, and I would like to conduct tests of equivalence between groups. For normal data, there are techniques like TOST (two one-sided t-tests). Is there anything analogous to TOST for non-normal data?

Answer 1

The logic of TOST employed for Wald-type t and z test statistics (i.e. $\theta / s_{\theta}$ and $\theta / \sigma_{\theta}$, respectively) can be applied to the z approximations for nonparametric tests like the sign, sign rank, and rank sum tests. For simplicity I assume that equivalence is expressed symmetrically with a single term, but extending my answer to asymmetric equivalence terms is straightforward.

One issue that arises when doing this is that if one is accustomed to expressing the equivalence term (say, $\Delta$) in the same units as $\theta$, then the the equivalence term must be expressed in units of the particular sign, signed rank, or rank sum statistic, which is both abstruse, and dependent on N.

However, one can also express TOST equivalence terms in units of the test statistic itself. Consider that in TOST, if $z = \theta/\sigma_{\theta}$, then $z_{1} = (\Delta - \theta)/\sigma_{\theta}$, and $z_{2} = (\theta + \Delta)/\sigma_{\theta}$. If we let $\varepsilon = \Delta / \sigma_{\theta}$, then $z_{1} = \varepsilon - z$, and $z_{2} = z + \varepsilon$. (The statistics expressed here are both evaluated in the right tail: $p_{1} = \text{P}(Z > z_{1})$ and $p_{2} = \text{P}(Z > z_{2})$.) Using units of the z distribution to define the equivalence/relevance threshold may be preferable for non-parametric tests, since the alternative defines the threshold in units of signed-ranks or rank sums, which may be substantively meaningless to researchers and difficult to interpret.

If we recognize that (for symmetric equivalence intervals) it is not possible to reject any TOST null hypothesis when $\varepsilon \le z_{1-\alpha}$, then we might proceed to make decisions on appropriate size of the equivalence term accordingly. For example $\varepsilon = z_{1-\alpha} + 0.5$.

This approach has been implemented with options for continuity correction, etc. in the package tost for Stata (which now includes specific TOST implementations for the Shapiro-Wilk and Shapiro-Francia tests), which you can access by typing in Stata:

Edit: Why the logic of TOST is sound, and equivalence test formations have been applied to omnibus tests, I have been persuaded that my solution was based on a deep misunderstanding of the approximate statistics for the Shapiro-Wilk and Shapiro-Francia tests

Answer 2

It's not a TOST per se, but the Komolgorov-Smirnov test allows one to test for the significance of the difference between a sample distribution and a second reference distribution you can specify. You can use this test to rule out a specific kind of different distribution, but not different distributions in general (at least, not without controlling for error inflation across tests of all possible alternatives...if that's somehow possible itself). The alternative hypothesis for any one test will remain the less specific "catch-all" hypothesis, as usual.

If you can settle for a test of distributional differences between two groups where the null hypothesis is that the two groups are equivalently distributed, you can use the Komolgorov-Smirnov test to compare one group's distribution to the other group's. That's probably the conventional approach: ignore the differences if they're not statistically significant, and justify this decision with a test statistic.

In any case, you may want to consider some deeper issues arising from the "all-or-nothing" approach to rejecting a null hypothesis. One such issue is very popular here on Cross Validated: "Is normality testing 'essentially useless'?" People like to answer normality-testing questions with a question: "Why do you want to test this?" The intention, I assume, is generally to invalidate the reason for testing, which may ultimately lead in the right direction. The gist of useful responses to the question I've linked here seems to be as follows:

1. If you're concerned about violations of parametric test assumptions, you should just find a nonparametric test that doesn't make distributional assumptions instead. Don't test whether you need to use the nonparametric test; just use it!
2. You should replace the question, "Is my distribution significantly non-normal?" with, "How non-normal is my distribution, and how is this likely to affect my analyses of interest?" For instance, tests regarding central tendency (especially involving means) may be more sensitive to skewness than to kurtosis, and vice versa for tests regarding (co)variance. Nonetheless, there are robust alternatives for most analytic purposes that aren't very sensitive to either kind of non-normality.

If you still wish to pursue a test of equivalence, here's another popular discussion on Cross Validated that involves equivalence testing.

Answer 3

Equivalence is never something we can test. Think about the hypothesis: $\mathcal{H}_0: f_x \ne f_y$ vs $\mathcal{H}_1: f_x = f_y$. NHST theory tells us that, under the null, we can choose anything under $\mathcal{H}_0$ that best fits the data. That means we can almost always get arbitrarily close to the distribution. For instance, if I want to test $f_x \sim \mathcal{N}(0, 1)$, the probability model that allows for separate distributions of $\hat{f}_x$ and $\hat{f}_y$ will always be more likely under the null, a violation of critical testing assumptions. Even if the sample $X=Y$ identically, I can get a likelihood ratio that is arbitrarily close to 1 with $f_y \approx f_x$.

If you know a suitable probability model for the data, you can use a penalized information criterion to rank alternate models. One way is to use the BICs of the two probability models (the one estimated under $\mathcal{H}_0$ and $\mathcal{H}_1$. I've used a normal probability model, but you can easily get a BIC from any type of maximum likelihood procedure, either by hand or using the GLM. This Stackoverflow post gets in nitty-gritty for fitting distributions. An example of doing this is here:

set.seed(123)
p <- replicate(1000, { ## generate data under the null
  x <- rnorm(100)
  g <- sample(0:1, 100, replace=T)
  BIC(lm(x~1)) > BIC(lm(x~g))
})
mean(p)

gives

> mean(p)
[1] 0.034

$p$ here is the proportion of times that the BIC of the null model (separate models) is better (lower) than the alternative model (equivalent model). This is remarkably close to the nominal 0.05 level of statistical tests.

On the other hand if we take:

set.seed(123)
p <- replicate(1000, { ## generate data under the null
  x <- rnorm(100)
  g <- sample(0:1, 100, replace=T)
  x <- x + 0.4*g
  BIC(lm(x~1)) > BIC(lm(x~g))
})
mean(p)

Gives:

> mean(p)
[1] 0.437

As with NHST there are subtle issues of power and false positive error rates that should be explored with simulation before making definitive conclusions.

I think a similar (perhaps more general method) is using Bayesian stats to compare the posterior estimated under either probability model.