# equivalence test - why isn't it more common?

Often in frequentist hypothesis testing, the null hypothesis is of the form: $H_0: \theta = 0$

I've seen many posts about how the p-value when doing tests against this null hypothesis is just a measure of sample-size in some sense, since in reality theta is almost never exactly 0, so given enough data points the p-value will converge to 0.

So then, why is there all this focus on that simple null hypothesis if we almost always know a priori that $\theta \neq 0$?

It seems very intuitive that the null should be something more like: $H_0: |\theta| < \epsilon$, for some $\epsilon$. Even if $\epsilon$ is small, this already seems like a much more applicable test to real data.

I've been learning about A/B testing and haven't seen the concept of equivalence testing brought up, yet it seems like a very natural concept to use in that setting? Is there some flaw in this methodology that I'm missing / is there any reason to prefer testing: $H_0: \theta = 0$?

edit: found a relevant comment by Keith Winstein from another thread (quoted below): Are large data sets inappropriate for hypothesis testing?

The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.

But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than αα (the tests's bound on false positives).

The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the αα, usually 0.05).

Again, makes perfect sense. What's the point of testing something like "probability exactly equal to 0.5" when we already know it's false ({0.5} is a set of measure 0)? Yet I see this done 1000000x more than tests of compounded hypothesis like Keith describes. The only things I've found about non-inferiority testing have been in the medical literature. I don't even remember learning about it in intro stats courses even though it's a very intuitive concept. Is it not more widely applicable?

• In the health field non-inferiority tests are quite common where the new (cheaper) method is accepted if it is not much worse than the (expensive) standard. See [this paper}(trialsjournal.biomedcentral.com/articles/10.1186/…) which is open access for some more detail. Commented Oct 1, 2016 at 9:09
• Thanks for the excellent link! I guess while I understand introducing hypothesis testing against the null H_0 : theta = 0 makes sense for pedagogical reasons or in very special cases in which theta can actually be 0, what's the point of using it other cases? For example, in lots of research people perform some regression and look for "statistically significant" variables, but here statistical significance is based on rejecting the null of H_0: theta = 0. Why not perform inferiority tests on the variables instead? Commented Oct 1, 2016 at 10:54