Experimental Design for Equivalence Testing

Question

I'm working on setting up a study that will test how quickly users click on various icons appearing on a computer screen using three different prototype mouse types. Subjects will visit on three three days and perform the test twice each visit (two recorded time measurements per visit).

I want to show there is no difference in time within subjects and between mouse devices via equivalence testing.

Let's say I have 135 subjects. I could assign 45 subjects to a specific mouse device and only have them use that particular device each visit. Or I could assign them a different mouse device each visit.

Which design would be better? I believe it depends on the variability between mouse devices. Also, how would I analyze the equivalence testing in one model that takes into account the user effect and mouse device effect?

Answer 1

"Usual" significance testing uses null hypothesis $H_0:|\mu_A-\mu_B|=0$. That is, we assume no effect exists. The "acceptance region" (AR, the region in which we do not reject the null) size shrinks as sample size grows. This leads to a super sensitive test, which will reject the null for tiny differences in large sample size.

Using indifference region is a somewhat more "realistic" approach: we acknowledge the possibility that some effect with exist, but we decide to ignore it up to a point, say we ignore effects smaller than $\epsilon$. A relevant null hypothesis would be $H_0:|\mu_A-\mu_B|<\epsilon$. It does help us achieve a more stable approach (in terms of false rejection of the null), but a large enough sample size would eventually lead to rejecting the null. See Section 4.3 of Bickel & Doksum for more details. Do note that the ignorable effect size should be determined before you conduct the test itself - it should be based on results of previous experiments, known measuring error, relevant literature and so on.

Equivalence testing is "flipping" the direction of the hypotheses, i.e $H_0:|\mu_A-\mu_b|\ge\epsilon$. the result is an AR which grows with sample size (up to a bound). The so called "problem" is that it requires a large enough sample size in order to have a large enough AR. Note that for ET, "acceptance" means that we do reject the relevant null htpothesis.

For example, consider the following problem: we have $n$ measurements of $X_A\sim N(\mu_A,1),X_B\sim N(\mu_B,1)$ and we hypothesize the difference of means $\mu_A-\mu_B$. The null hypotheses are as follows:

$$H_0^{sig}:|\mu_A-\mu_B|=0,\qquad H_0^{indiff}:|\mu_A-\mu_B|<\epsilon,\qquad H_0^{equiv}:|\mu_A-\mu_B|>\epsilon$$

We take a vector of size 100, $X\sim N_{100}(0, I)$ and repeat it $2^k$ times. See how the AR size changes:

^{(This plot is taken from the background section of my MA thesis).}

For your case, I believe that ET wouldn't be very helpful as the sample size isn't that large. Try using indifference region and see how it goes.