equivalence test - why isn't it more common?

Question

Often in frequentist hypothesis testing, the null hypothesis is of the form: $H_0: \theta = 0$

I've seen many posts about how the p-value when doing tests against this null hypothesis is just a measure of sample-size in some sense, since in reality theta is almost never exactly 0, so given enough data points the p-value will converge to 0.

So then, why is there all this focus on that simple null hypothesis if we almost always know a priori that $\theta \neq 0$?

It seems very intuitive that the null should be something more like: $H_0: |\theta| < \epsilon$, for some $\epsilon$. Even if $\epsilon$ is small, this already seems like a much more applicable test to real data.

I've been learning about A/B testing and haven't seen the concept of equivalence testing brought up, yet it seems like a very natural concept to use in that setting? Is there some flaw in this methodology that I'm missing / is there any reason to prefer testing: $H_0: \theta = 0$?

edit: found a relevant comment by Keith Winstein from another thread (quoted below): Are large data sets inappropriate for hypothesis testing?

The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.

But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than αα (the tests's bound on false positives).

The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the αα, usually 0.05).

Again, makes perfect sense. What's the point of testing something like "probability exactly equal to 0.5" when we already know it's false ({0.5} is a set of measure 0)? Yet I see this done 1000000x more than tests of compounded hypothesis like Keith describes. The only things I've found about non-inferiority testing have been in the medical literature. I don't even remember learning about it in intro stats courses even though it's a very intuitive concept. Is it not more widely applicable?

Answer 1

I did some more thinking on the consequences of using a "point" null hypothesis. I plotted some power curves below:

Basically for a fixed alpha when doing the usual point null hypothesis, as n increases, the power for small true parameters increases. This just doesn't seem like a great property to me in many situations in which we want to decide whether the true effect is "meaningful" (i.e practical /economic significance vs statistical significance).

For example, imagine an academic journal accepting entries on the basis of "statistical significance." Even if it has power requirements, it'll tend to just end up with papers having a lot of small true effects, not very interesting.

In contrast, the power curves for the "two one sided tests" constructed below become better suited to their purpose as n-increases. If we fix Power(epsilon) = 0.05, as n increases Power(epsilon + delta) -> 1 for a small delta. A journal that accepted entries on this rule would (hypothetically) end up with lots of results that are actually economically significant (obviously I'm assuming the ideal case in which researchers aren't doing multiple hypothesis testing, are preregistering experimental design, etc...)