1
$\begingroup$

In a previous post that I found very informative, it was claimed that:

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero...

Is this only true when conducting two-sided tests? I think that if you perform a one-sided test, this may not always be true.

If my intuition is correct, is it best practice to have a hypothesis of the direction of the effect, and favor performing one-sided tests? I understand that some claim one-sided tests have less power than two-sided tests, but you can easily adjust the significance level to account for that (i.e., instead of running a two-sided test at the 95% level, run a one-sided test at the 97.5% level).

$\endgroup$
4
$\begingroup$

Yes, your interpretation of the statement is correct. Such a test will always show significant results if it is two-sided but this is not true if it is one-sided.

I don't agree with your suggested implications, though. It treats the hypothesis test as the goal of the research project, which it shouldn't be.

In my opinion, best practice would be to avoid testing simple null hypothesis that an effect size is precisely zero, because those are almost never true or informative. My preference would be to focus on developing theory that makes quantitative predictions. This theory development need not be only done with pure math - experiments, observations and exploratory analysis all have important roles to play. Once a useful theory is sufficiently developed, test the specific quantitative predictions that it makes.

I recognise that this is idealised and not always feasible. But since you ask for best practice, I'd advocate this broad approach.

$\endgroup$
  • $\begingroup$ The intuition is that, if you're assuming that something is exactly zero, look at how many (infinitely) more possibilities there are. However, if you assume negative, there are as many possibilities of negative values as positive values. $\endgroup$ – Dave Aug 6 '19 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.