6
$\begingroup$

(Sorry if this is obvious or is a duplicate. I couldn't find one.)

Suppose two researchers are studying whether average height of some population has changed significantly. Researcher 1 hypothesizes that there has been some change (two-tailed) and researcher 2 hypothesizes that it has increased (one-tailed). Both want a p-value < 0.05 (as seems to be common in publications).

Researcher 2 will have a lower critical value, so she may be able to reject the null hypothesis when researcher 1 cannot. So now we have one study showing that heights have significantly increased, and another (using the same data and p-value) showing that they haven't changed.

Is that weird? Am I thinking about it wrong? Did I flub something? Even if it's totally correct, wouldn't it lead to misunderstandings? "Studies [on the same data] show that heights have not changed, but have gone up."

Improve this question
$\endgroup$
6
  • $\begingroup$ You're correct. Statistics is weird like that. Many things have to be derived very carefully and mathematically, and at the end of the day the conclusion you draw is never truly black or white. $\endgroup$
    – KenHBS
    Aug 18 '17 at 17:16
  • $\begingroup$ Your example supports the use of effect sizes to understand or interpret your findings. Personally, I find both effect sizes and statistical significance useful. $\endgroup$
    – Joel W.
    Aug 18 '17 at 18:42
  • $\begingroup$ Researcher 2 is making a different assumption and has a different objective than Researcher 1. Contrary to the impression given by @KenS (who very well might be correct in general about statistics being unintuitive), it seems perfectly sensible to me--and intuitively obvious--that the two could (and, on occasion, should) arrive at different conclusions in the same circumstances. It would be weird if some theory required both of them always to draw the same conclusions from the same data--that ought to lead us to suspect the theory was flawed. $\endgroup$
    – whuber ♦
    Aug 18 '17 at 19:09
  • $\begingroup$ Thanks. Perhaps my question is a more sociological one. I edited my question to be clear about the weird part: a journalist (or other layperson) seems to be able to conclude "studies [on the same data] show that heights have not changed, but have gone up." $\endgroup$
    – monk
    Aug 18 '17 at 20:07
  • $\begingroup$ This is understandable to US citizens, who at a very early age are taught the distinction between "innocent until proven guilty" and "presumed guilty until shown innocent." It is clear even to children (and I don't mean to cast any aspersions in saying so) that in one situation a person will not be convicted but in the other they can be, even when the evidence and the arguments in both cases are identical. This is so close in spirit to how statistical testing works that it is often invoked as an analogy. Perhaps, as an analogy, it could help your journalist craft an explanation. $\endgroup$
    – whuber ♦
    Aug 18 '17 at 20:51
4
$\begingroup$

Case 1: Entertaining the hypothesis that average height may have increased or decreased, we cannot reject the null hypothesis that neither has happened.

Case 2: Entertaining the hypothesis that average height may have increased, we reject the null hypothesis that it hasn't.

Both examined at the same accepted Type I error probability. (e.g. 5%).

By "casting a wider net" (Case 1), we require more from our data sample, since we ask from it to statistically "disprove/not disprove" two effects at once (increase-decrease).

Assume that descriptive statistics of the data sample indicate that the current average height is greater than in the past. The data is already showing us the way, and what is left is to test statistically whether the observed increase is statistically large enough. To execute here a two-tailed test would be wrong, since it would artificially dilute the informational potential of the data sample.

Improve this answer
$\endgroup$
2
$\begingroup$

As @Aksakal says, there is nothing weird about this: it is easy to see that the significance level (for a continuous random variable) is equal to the probability of a type I error.

So your one-sided and two sided test have the same type I error probability. What differs is the power of the two tests. If you know that the alternative is an increase, then for the same type I error probability, the type II error probability is lower with the one sided test (or the power is higher).

In fact, it can be shown that, for a given type I error probability (and in the univariate case), the one sided test is the most powerfull you can find, whatever the alternative is. This is thus the UMPT, the Uniformly Most Powerful Test.

It all depends on what you want to test. Assume you want to buy lamps from your supplier and the supplier says that the life time of a lamp is 1000 hours (on average). If you want to test these lamps then you will probably not care if these lamps live longer so you will test $H_0: \mu=1000$ versus $H_1: \mu < 1000$ because this test, for the same type I error probability, has more power.

see also What follows if we fail to reject the null hypothesis?

Improve this answer
$\endgroup$
2
$\begingroup$

There is nothing weird about this results. The reason why this result looks weird to you is because you're using the same significance level.

The hypo 1 includes the possibility that the height either went down or up, while hypo 2 only includes the increase. So, intuitively (but not precisely) you need to compare the critical values of 0.05 significance of hypo 2 and 0.1 significance of hypo 1.

Again, don't take these literally, this is just to point out that you can't compare the critical values at the same significance of these hypos.

UPDATE: you journalist should not be reporting on the statistical studies if she can't interpret them. There's no other way about this. Writing "Studies [on the same data] show that heights have not changed, but have gone up" simply disqualify the person from her job.

Improve this answer
$\endgroup$
1
$\begingroup$

In many fields (e.g. medical statistics where you may be comparing a new drug vs. an old one) the convention is that one-sided tests are done at 2.5% by default (vs. two-sided ones at 5%). And that, even if you only hypothesize an effect in direction, two-sided tests are done. This convention has in part developed to prevent people from switching to one-sided tests just to increase type 1 errors in the desired direction.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.