How to perform a ANOVA for statistical equivalence?

I red about the two one-sided test (TOST) for equivalence, but (I think) for this study design it is not possible to perform a classical t-test, so a repeated measures ANOVA is needed.

The study design is a common pre-post treatment-control design: There are two different groups (control / treatment) and dependent two measure points at the baseline and a followup (MP1 / MP2). The research question is, if the treatments are equal.

My thought is, that the ANOVA analyses the differences between the groups. Is there a post-hoc for equivalence, special in R?


Thanks to D_Williams, I red the using-lsmeans vignette and the lsmeans reference manual. There is a function called test. With this function you can do equivalence / noninferiority or nonsuperiority tests. Now I got stuck with a new problem. Because of the study design I need a linear regression with repeated measures. So here is a example dataset:

 dataset <- data.frame (ID    = rep(1:16),
                  GROUP = factor(rep(c("A","B"),8)),
                   MP1   = c(15,12,20,17,28,24,17,10,14,10,25,23,9,18,19,20),
                   MP2   = c(12,9,19,10,20,15,12,5,12,10,22,15,8,17,10,19),

The linear regression should be:

data.lm <- lm (MP2 - MP1 ~ GROUP, data=dataset)

As far as I understood this answer right, Case 2b is the correct one for me. Because there are two quantitative variables and a one dichotomous variable.

Afterwards I run these commands:

data.lsm <- lsmeans(data.lm,"GROUP")
test(data.lsm, null = log(100), delta = 0.20, side="equivalence")

Delta is the equivalence margin or the range of similarity. But I am not sure what the log(100) is.

My questions are:

  1. Is this approach right?


  2. To be honest: I don't really understand the last steps. Does anybody got some code example for studying them?

  • $\begingroup$ What are MP1 & MP2 ? Are the control and treatment group made by the same units? $\endgroup$ – utobi Nov 12 '16 at 11:45
  • $\begingroup$ MP1 and MP2 are the two dependent measure points at the beginning and at the end. The groups are not in the same unit. $\endgroup$ – M.Unterreiner Nov 14 '16 at 8:33
  • $\begingroup$ Ok, but what do you mean by "equivalence" test? Are you interested in the mean/median values or on the entire distribution of MP1&MP2. Just to be sure I understood correctly, before answering to your question. $\endgroup$ – utobi Nov 15 '16 at 12:52
  • $\begingroup$ Usually you test for superiority with the H1 hypothesis testing. Here, the hypothesis H1 is, that the control treatment and the "new" treatment is equal. In this case I am interested in the mean / medians values. $\endgroup$ – M.Unterreiner Nov 15 '16 at 13:43
  • $\begingroup$ @M.Unterreinter I think you are making confusion about H0 and H1 hypothesis. Typically, H0 is the equality hypothesis and H1 is the alternative, which can be uni or two-directional. In your case, I suspect, you have H0: old treatment = new treatment, H1: old treatment different from new treatment (or old treatment worse than new treatment ). Is that so? $\endgroup$ – utobi Nov 15 '16 at 13:50

Since you have only two groups, there is no need to perform an ANOVA. You can perform a TOST procedure by simply building a confidence interval (CI) for a two sample problem, say x and y, where x = MP2-MP1 in the control group and y = MP2-MP1 in the treated group. You need to pay attention to the usual assumptions for the statistical tests: normality, heteroskedasticity, etc. But, once you have your CI, you can see if it is contained within the +-delta region or not. In R you can do something like this:

x = rnorm(100)
y = rnorm(70)

# for two sample t-test with equal variances
tt <- t.test(x, y, var.equal = TRUE, conf.level = 0.90)

# for Welch two sample t-test
tt.uneq <- t.test(x, y, var.equal = FALSE, conf.level = 0.90)

# for two sample Wilcoxon test
wcox <- wilcox.test(x,y, conf.int = TRUE, conf.level = 0.90)

delta <- 0.32

plotCI(x = 1, y=diff(rev(tt$estimate)), ui = tt$conf.int[2], li = tt$conf.int[1],
       xlim=c(0,4), ylab = "Confidence intervals", ylim = c(-0.6, 0.6),
       xlab ="Methods")
plotCI(x = 2, y=diff(rev(tt.uneq$estimate)), ui = tt.uneq$conf.int[2],
       li = tt.uneq$conf.int[1], add=TRUE, col=2)
plotCI(x = 3, y=wcox$estimate, ui = wcox$conf.int[2],
       li = wcox$conf.int[1], add=TRUE, col=3)
abline(h = c(-delta, delta), lwd = 2, lty = 2)

enter image description here

Here, to illustrate the idea I'm considering three type of tests but there are many others in the literature. Hope this helps.

| cite | improve this answer | |
  • $\begingroup$ Thank you really much utobi. I thought that you can't use a t-test with repeated measurements and two groups because of the Type I error. Is it right, that with a CI there won't be this kind of error? $\endgroup$ – M.Unterreiner Nov 16 '16 at 20:10
  • $\begingroup$ M.Unterreiner, the type I error rate is ubiquitous in statistical testing. Having said that, nothing special with repeated measures. In such a case, you have balanced samples so you can use either t.test or wilcox.test but you have to specify the option "paired=TRUE". $\endgroup$ – utobi Nov 17 '16 at 8:29
  • $\begingroup$ What if you also wanted to test that the increases (MP2-MP1) for each condition were significant? ie. MP2 was significantly different from MP1 in each condition. How would you test for all of these without inflating your Type-1 error rate? $\endgroup$ – strava Jul 26 at 19:21

Generally equivalence testing is very limited to simple models. I have, however, recently learned that the lsmeans package in r does equivalence testing for models more complex than a t-test. I have not used it, but I have used lsmeans and it is a very flexible tool. This might work, but you will have to check out the package vignette for the details to be sure see here

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.