Often in frequentist hypothesis testing, the null hypothesis is of the form: $H_0: \theta = 0$

I've seen many posts about how the p-value when doing tests against this null hypothesis is just a measure of sample-size in some sense, since in reality theta is almost never exactly 0, so given enough data points the p-value will converge to 0.

So then, why is there all this focus on that simple null hypothesis if we almost always know a priori that $\theta \neq 0$?

It seems very intuitive that the null should be something more like: $H_0: |\theta| < \epsilon$, for some $\epsilon$. Even if $\epsilon$ is small, this already seems like a much more applicable test to real data.

I've been learning about A/B testing and haven't seen the concept of equivalence testing brought up, yet it seems like a very natural concept to use in that setting? Is there some flaw in this methodology that I'm missing / is there any reason to prefer testing: $H_0: \theta = 0$?

edit: found a relevant comment by Keith Winstein from another thread (quoted below): Are large data sets inappropriate for hypothesis testing?

The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.

But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than αα (the tests's bound on false positives).

The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the αα, usually 0.05).

Again, makes perfect sense. What's the point of testing something like "probability exactly equal to 0.5" when we already know it's false ({0.5} is a set of measure 0)? Yet I see this done 1000000x more than tests of compounded hypothesis like Keith describes. The only things I've found about non-inferiority testing have been in the medical literature. I don't even remember learning about it in intro stats courses even though it's a very intuitive concept. Is it not more widely applicable?

  • $\begingroup$ In the health field non-inferiority tests are quite common where the new (cheaper) method is accepted if it is not much worse than the (expensive) standard. See [this paper}(trialsjournal.biomedcentral.com/articles/10.1186/…) which is open access for some more detail. $\endgroup$ – mdewey Oct 1 '16 at 9:09
  • $\begingroup$ Thanks for the excellent link! I guess while I understand introducing hypothesis testing against the null H_0 : theta = 0 makes sense for pedagogical reasons or in very special cases in which theta can actually be 0, what's the point of using it other cases? For example, in lots of research people perform some regression and look for "statistically significant" variables, but here statistical significance is based on rejecting the null of H_0: theta = 0. Why not perform inferiority tests on the variables instead? $\endgroup$ – convolutedstatistic Oct 1 '16 at 10:54

I did some more thinking on the consequences of using a "point" null hypothesis. I plotted some power curves below:

enter image description here

Basically for a fixed alpha when doing the usual point null hypothesis, as n increases, the power for small true parameters increases. This just doesn't seem like a great property to me in many situations in which we want to decide whether the true effect is "meaningful" (i.e practical /economic significance vs statistical significance).

For example, imagine an academic journal accepting entries on the basis of "statistical significance." Even if it has power requirements, it'll tend to just end up with papers having a lot of small true effects, not very interesting.

In contrast, the power curves for the "two one sided tests" constructed below become better suited to their purpose as n-increases. If we fix Power(epsilon) = 0.05, as n increases Power(epsilon + delta) -> 1 for a small delta. A journal that accepted entries on this rule would (hypothetically) end up with lots of results that are actually economically significant (obviously I'm assuming the ideal case in which researchers aren't doing multiple hypothesis testing, are preregistering experimental design, etc...)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.