# What is the best method of reporting multiple tests of equivalence?

I am doing a study which will involve multiple tests of equivalence. Is there a standard table for reporting such results?

EDIT with more detail: It is a longitudinal study with 5 time points. Our hypothesis is that there is a change from baseline to 1 month and then stability after that. Initially, I thought of a multilevel model but there is no independent variable other than time. So I decided on t-tests between baseline and 1 month and TOST tests of equivalence between 1 and 3 months, 3 and 6, 6 and 9 and 9 and 12.

There were also 5 areas tested.

So, I need a way to summarize all these tests in a table

• Peter can you say a little more? Number of tests? Independent or dependent tests? Also: are you defining equivalence in units of your measure, or in units of your test statistic? Does this definition change for each test, or is it the same for each test? FWIW I simply report multiple equivalence tests the same way I report regression output: tabled, with one test per row... if you are also performing tests for difference, you can side-by-side the results this way in separate columns to facilitate "relevance testing" results. – Alexis Mar 5 '16 at 0:35
• Thanks for these clarifications. I cannot see why an LME would not work. It seems almost natural for this. Please see my answer below where I tried to simulate something based on your description. – usεr11852 says Reinstate Monic Mar 6 '16 at 0:44
• In case you wish to use t-test, I recommend that you go with approximate randomization instead. It has either equal or lower type I or II errors. It also spits out the $p$ values right away. – caveman Mar 8 '16 at 5:16
• Peter, I have a question that was not touched by any of the answers here: As you may know, we can use a 90%-CI ($1-2\alpha$) to perform a TOST at the 95%-level: if the CI is inside the equivalence region, we conclude equivalence. In the case of multiple TOSTs as in your case, could you adjust the confidence level for multiple comparisons? So in your case of 4 TOSTs, we would use a conservative $\alpha$ of $0.05/4 = 0.0125$ which would correspond to $1-2\alpha = 1 - 2\cdot 0.0125 = 0.975$, so 97.5%-level confidence intervals for the 4 TOST? Does that sound reasonable to you? – COOLSerdash Apr 17 '19 at 7:50
• @COOLSerdash Certainly. And I think it raises the same questions of multiple comparison adjustment in the more usual methods. – Peter Flom - Reinstate Monica Apr 17 '19 at 11:29

Regression table presentations are easy enough to modify to accommodate tests for equivalence, including relevance tests—where you base conclusions off of both tests for difference (tests of $H^{^{+}}_{0}$) and tests for equivalence (tests of $H^{^{-}}_{0}$). For example (assuming you are presenting multiple tests in a regression context, hence the $\beta$):

You can present both one-sided test statistics ($t_1$ and $t_2$) and their associated p-values ($p_1$ and $p_2$) from the tests for equivalence, and in addition present the test statistic $t$ and p-value ($p$) for the tests of difference.

In addition you may want to include a column for your definition of equivalence if it varies from test to test (I use $\Delta$ to indicate my equivalence/relevance threshold defined in units of my measures, and $\varepsilon$ to indicate this threshold defined in units of my test statistic). If you use a consistently defined of equivalence/relevance threshold for all tests, you would likely indicate that in a footnote to the table.

You can also facilitate interpretation by including columns to explicitly articulate rejection decisions for equivalence and difference tests. Including a relevance test column (combining results as I have illustrated here) may also facilitate interpretation.

Of course, one can also use this format to present independent tests, and to present different kinds of TOST test statistics (e.g. z test statistics like those used with non-parametric tests, exact binomial test statistics, etc.).

I think one can do all these multiple tests of equivalence within a single linear-mixed models. Given you have multiple (2+) measures after the change took place it is rather natural to present these multiple tests as part of a single repeated-measurements model.

In particular, one could define indicator variables between the successive steps and then check their significance; essentially doing multiple $t$-test in one-go. I think that a random structure with a simple intercept and slope for each subject would be fine. I do not see the absence of independent variables other than time as a structural problem. If anything I think it simplifies matters further.

From what I understand given a starting value(val0) something takes place (step0) while stepping from the first measurement period to the second. For the subsequent intra-measurement time periods (step1, step2, step3) nothing happens. Measurement error is assumed constant. So one has something like this:

I create this sample with the following code:

set.seed(123)
sampleTimes <- seq(0,1, length.out = 5);
N = 10^2;
val0 <- rnorm(N, mean = 0, sd = 5); # Starting values
slopeAt0 <- rnorm( N, mean = -10, sd = 5); # Effect kicks in
val1to5 <- val0 + slopeAt0 * diff(sampleTimes[1:2]) # so val0 is +2.5 higher

trueMeans <- cbind( val0, t(matrix(rep(val1to5,4), 4, byrow = TRUE)))
obsSample <- trueMeans + rnorm(N*5)
subject <- (1:N)
matplot(sampleTimes, t(obsSample),type = 'l', ylab= 'Obs. Sample') # Visualise


And define a series of indicator variables for the periods stepping from one measurement point to the next. Notice that I define no "last step" step4; we do not know what happens after the last measurement point at $t_4$.

Q <- data.frame( t = rep(sampleTimes, times = N),
ID = rep(subject, each = 5), reads = as.vector(t(obsSample)),
step0 = rep(c(1,0,0,0,0), times = N),
step1 = rep(c(0,1,0,0,0), times = N),
step2 = rep(c(0,0,1,0,0), times = N),
step3 = rep(c(0,0,0,1,0), times = N))


Using this design it is a relatively simple task to fit an LME and check if the stepX variables become statistically significant. The intercept and slope will absorb any subject-specific variations and one bootstrap that model directly. One can also use the $t$-values from the original LME too.

library(lme4)
m1 <- lmer(reads ~ step0 + step1 + step2 + step3 + (t+1|ID), Q)
summary(m1)
confZ = confint(m1, method='boot', nsim= 1000)
print(confZ)
# Computing bootstrap confidence intervals ...
#                 2.5 %     97.5 %
# .sig01       3.7653500  5.0961341
# .sig02      -0.2043878  0.9843421
# .sig03       0.3396433  1.2702728
# .sigma       0.9869698  1.1579640
# (Intercept) -3.2169570 -1.2982469
# step0        2.5430254  3.2526163
# step1       -0.2465836  0.4475912
# step2       -0.0728649  0.5659132
# step3       -0.1625806  0.4023341


The results are quite reasonable I think, even for the modest sample size ($N = 10^2$) used. Wanting to be on the safe side I included a random slope and this came as probably redundant (sig02) (the original problem description does not specify it there is an additional time-varying trend) but excluding it does not alter the basic findings in any way: something happens during the step0 period.

Others have given you more direct answers to your question, but I'm going to try and show a different solution to the problem. Unless I'm misunderstanding you, it seems like a two-way (period area) fixed effects model would work best (with robust SEs, of course!). $$y_i = \sum_k \alpha_k \times \mathbf{1}[i \in \text{period}_k] + \sum_j \beta_j \times \mathbf{1}[i \in \text{area}_j] + \epsilon_i$$

You could do an F-test (joint null test) on all the $\alpha_k$ to see if there is any change across time at all, and you can do a Wald test to see if a subset of coefficients (in your case $\alpha_2 = \alpha_3 = \dots = \alpha_K$) are the same. This also gets around all of the multiple testing problems you should be worrying about if you're doing a bunch of pairwise tests.

All of the above can be done within the text. For a visual inspection.. I know you're asking for a table, but I think a plot like this with 95% intervals is much more convincing. But maybe it's just the style of my field.