How to perform a ANOVA for statistical equivalence?

I read about the two one-sided test (TOST) for equivalence, but (I think) for this study design it is not possible to perform a classical t-test, so a repeated measures ANOVA is needed.

The study design is a common pre-post treatment-control design: there are two different groups (control / treatment) and dependent two measure points at the baseline and a followup (MP1 / MP2). The research question is, if the treatments are equal.

My thought is, that the ANOVA analyses the differences between the groups. Is there a post-hoc for equivalence, special in R?


Thanks to D_Williams, I read the using-lsmeans vignette and the lsmeans reference manual. There is a function called test. With this function you can do equivalence / noninferiority or nonsuperiority tests. Now I got stuck with a new problem. Because of the study design I need a linear regression with repeated measures. So here is a example dataset:

 dataset <- data.frame (ID    = rep(1:16),
                  GROUP = factor(rep(c("A","B"),8)),
                   MP1   = c(15,12,20,17,28,24,17,10,14,10,25,23,9,18,19,20),
                   MP2   = c(12,9,19,10,20,15,12,5,12,10,22,15,8,17,10,19),

The linear regression should be:

data.lm <- lm (MP2 - MP1 ~ GROUP, data=dataset)

As far as I understood this answer right, Case 2b is the correct one for me. Because there are two quantitative variables and a one dichotomous variable.

Afterwards I run these commands:

library("lme4", "lsmeans", "estimablity")
data.lsm <- lsmeans(data.lm, "GROUP")
test(data.lsm, null = log(100), delta = 0.20, side="equivalence")

Delta is the equivalence margin or the range of similarity. But I am not sure what the log(100) is.

My questions are:

  1. Is this approach right?


  2. To be honest: I don't really understand the last steps. Does anybody have some code example for studying them?

  • $\begingroup$ What are MP1 & MP2 ? Are the control and treatment group made by the same units? $\endgroup$
    – utobi
    Nov 12, 2016 at 11:45
  • $\begingroup$ MP1 and MP2 are the two dependent measure points at the beginning and at the end. The groups are not in the same unit. $\endgroup$ Nov 14, 2016 at 8:33
  • $\begingroup$ Ok, but what do you mean by "equivalence" test? Are you interested in the mean/median values or on the entire distribution of MP1&MP2. Just to be sure I understood correctly, before answering to your question. $\endgroup$
    – utobi
    Nov 15, 2016 at 12:52
  • $\begingroup$ Usually you test for superiority with the H1 hypothesis testing. Here, the hypothesis H1 is, that the control treatment and the "new" treatment is equal. In this case I am interested in the mean / medians values. $\endgroup$ Nov 15, 2016 at 13:43
  • $\begingroup$ @M.Unterreinter I think you are making confusion about H0 and H1 hypothesis. Typically, H0 is the equality hypothesis and H1 is the alternative, which can be uni or two-directional. In your case, I suspect, you have H0: old treatment = new treatment, H1: old treatment different from new treatment (or old treatment worse than new treatment ). Is that so? $\endgroup$
    – utobi
    Nov 15, 2016 at 13:50

2 Answers 2


Since you have only two groups, there is no need to perform an ANOVA. You can perform a TOST procedure by simply building a confidence interval (CI) for a two sample problem, say x and y, where x = MP2-MP1 in the control group and y = MP2-MP1 in the treated group. You need to pay attention to the usual assumptions for the statistical tests: normality, heteroskedasticity, etc. But, once you have your CI, you can see if it is contained within the +-delta region or not. In R you can do something like this:

x = rnorm(100)
y = rnorm(70)

# for two sample t-test with equal variances
tt <- t.test(x, y, var.equal = TRUE, conf.level = 0.90)

# for Welch two sample t-test
tt.uneq <- t.test(x, y, var.equal = FALSE, conf.level = 0.90)

# for two sample Wilcoxon test
wcox <- wilcox.test(x,y, conf.int = TRUE, conf.level = 0.90)

delta <- 0.32

plotCI(x = 1, y=diff(rev(tt$estimate)), ui = tt$conf.int[2], li = tt$conf.int[1],
       xlim=c(0,4), ylab = "Confidence intervals", ylim = c(-0.6, 0.6),
       xlab ="Methods")
plotCI(x = 2, y=diff(rev(tt.uneq$estimate)), ui = tt.uneq$conf.int[2],
       li = tt.uneq$conf.int[1], add=TRUE, col=2)
plotCI(x = 3, y=wcox$estimate, ui = wcox$conf.int[2],
       li = wcox$conf.int[1], add=TRUE, col=3)
abline(h = c(-delta, delta), lwd = 2, lty = 2)

enter image description here

Here, to illustrate the idea I'm considering three type of tests but there are many others in the literature. Hope this helps.

  • $\begingroup$ Thank you really much utobi. I thought that you can't use a t-test with repeated measurements and two groups because of the Type I error. Is it right, that with a CI there won't be this kind of error? $\endgroup$ Nov 16, 2016 at 20:10
  • $\begingroup$ M.Unterreiner, the type I error rate is ubiquitous in statistical testing. Having said that, nothing special with repeated measures. In such a case, you have balanced samples so you can use either t.test or wilcox.test but you have to specify the option "paired=TRUE". $\endgroup$
    – utobi
    Nov 17, 2016 at 8:29
  • $\begingroup$ What if you also wanted to test that the increases (MP2-MP1) for each condition were significant? ie. MP2 was significantly different from MP1 in each condition. How would you test for all of these without inflating your Type-1 error rate? $\endgroup$
    – strava
    Jul 26, 2020 at 19:21

Generally equivalence testing is very limited to simple models. I have, however, recently learned that the lsmeans package in r does equivalence testing for models more complex than a t-test. I have not used it, but I have used lsmeans and it is a very flexible tool. This might work, but you will have to check out the package vignette for the details to be sure see here

As of 2022, the emmeans package (successor to lmmeans can be used for equivalence tests of linear models; vignette info is here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.