I'm trying to run an experiment to see if my test and control (in this case 2 survey types) are similar, as opposed to significantly different, from each other.

Is there a test/way to do this? almost like an inverse of a t-test?


It's a bit tricky but what you are looking for is called equivalence testing (see wiki).

The general idea is this: assume we have 2 surveys $A,B$. we would like to formulate our hypotheses over $\mu_A, \mu_B$. An initial significance testing hypotheses would be:


we assume the means are exactly the same, meaning their difference is 0:


and then you know what to do (t-test). The next step is defining an indifference region around the null hypothesis. That is, a region of size $\delta$ around the null hypothesis $[\mu_0-\delta,\mu_0+\delta]$ (which in our case is simply $[-\delta,\delta]$) that we are indifferent to. The idea is to tolerate results around the null just as if thy were the null, so we don't reject the null for minor differences. So our updated hypotheses are now:

$$H_0:|\mu_A-\mu_B|\leq \delta,~~~H_1:|\mu_A-\mu_B|>\delta$$

Notice that using absolute values requires a slight change to the statistical test. Still, our null hypothesis is that the means differ by at most $\delta$. It is up to the data (and the statistician) to prove that the means differ.

The final step here is moving to equivalence testing (AKA noninferiority). The most important thing is the conceptual change: rather than assuming by default (i.e the null) that the means are equal, we now assume that they differ. Not only we assume they differ, but the difference is at least $\delta$. We formulate our hypotheses as:

$$H_0:|\mu_A-\mu_B|\geq \delta,~~~H_1:|\mu_A-\mu_B|<\delta$$

So we have inverted the hypotheses, and now it is up to the data (and the statistician) to prove that the means do not differ.

This ain't a trivial move, and it requires choosing both $\delta$ and $\alpha$ prior to observing the data. I hope this answers your question.

  • 1
    $\begingroup$ Interesting answer! What about when the samples from $A$ and $B$ are not normally-distributed? $\endgroup$ Jul 21 at 9:21
  • 3
    $\begingroup$ If sample size is large enough you just use $\sqrt{n}(\bar{X}^A_n-\mu_A)/s_A$ which is $\sim N(0,1)$ due to CLT (however it does require that the first two moments of $A,B$ are finite). $\endgroup$
    – Spätzle
    Jul 21 at 9:34
  • 3
    $\begingroup$ +1 @AlexandruDinu There are tests for equivalence of non-normal data also. A simple application is for z tests for proportion difference, and these have been extended to z approximate tests for rank sum and sign rank tests for equivalence. There are also omnibus equivalence tests (e.g., all means are different by at least $\delta$), and others in the uniformly most powerful test approach to equivalence, but that math is fancier than for TOST. See Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (Second Edition). Chapman and Hall/CRC Press. $\endgroup$
    – Alexis
    Jul 21 at 20:22
  • $\begingroup$ @AlexandruDinu You can also combine tests for equivalence and tests for difference (and thereby side-step confirmation bias in hypothesis testing). See my comments about combined inference in relevance tests in my answer here. Note also that CV has a tag for the simple two one-sided tests approach to equivalence: [tost], & an [equivalence] tag. $\endgroup$
    – Alexis
    Jul 21 at 20:23
  • 1
    $\begingroup$ Ah, the Wellek book. Spent hundreds of unforgettable hours with it while writing my Master's thesis. $\endgroup$
    – Spätzle
    Jul 22 at 4:05

If this is the case, then I encourage you to determine the confidence interval of the mean difference (the narrower, the better) which will make your demonstration even stronger.

First, I would encourage you to visualize your sample data to check whether your assumption is likely or not. Then, one possibility could be to identify a 95% confidence interval (CI) for the difference of means (or even a 99% if you are sure that they are similar) , you can refer to the "basic steps" chapter of https://en.wikipedia.org/wiki/Confidence_interval. To demonstrate your point, your CI must include 0 and the lower and upper values of your CI must be very close to 0 (this depends of the unit/scale of your test of course). I hope it helps!

  • $\begingroup$ Hi Pitouille - thank you for the response. Can you expound on this a bit in simple terms? $\endgroup$
    – jc315
    Jul 21 at 7:38
  • $\begingroup$ just adapted my answer... I hope it reads better! $\endgroup$
    – Pitouille
    Jul 21 at 8:06
  • 1
    $\begingroup$ @Pitouille No, that is not how you interpret a CI. This approach that you mention is based solely on subjectively checking the CI, there is no testing here. $\endgroup$ Jul 21 at 9:48
  • 1
    $\begingroup$ I disagree that the confidence interval has to include zero. That seems to be using an insignificant hypothesis test as evidence in favor of the null. $\endgroup$
    – Dave
    Jul 21 at 9:54
  • 1
    $\begingroup$ @user2974951 absolutely! This is not a test. The answer from Spätzle is rather interesting. $\endgroup$
    – Pitouille
    Jul 21 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.