4
$\begingroup$

Imagine the following scenario:

H0: kids in a town sleep 7 hours on average (mu= 7) HA: kids in a town do not sleep 7 hours on average (mu=! 7)

And then you collect your sample of town kids and you get an estimate of the mean: 7.8 hours and a standard deviation of 1.0 hour.

Now, I don't really understand why we would perform a two-sided t-test.

The two-sided test seems to give answer on the question: "how big is the chance to get a result that is x distance and further from the assumed (H0) mean?". So, in our case: the chance of 7.8 hours and up + 6.2 hours and down.

I don't understand why we are interested in the "6.2 and down" part. Why don't we just want to know what the chance is on just the outcome we got and more extreme, assuming the H0 is true?

After searching the internet for answers, I read that "you can't switch from a two-sided t-test to a one-sided t-test, because that's cheating". I still don't really understand why. All I understand so far is "if you choose a test based on the data, you might end up "overfitting" the test on your data". But I'm not sure if this analogy with statistical models is correct at all.

Thank you so much for reading!

$\endgroup$

4 Answers 4

12
$\begingroup$

Look at your hypotheses:

H0: kids in a town sleep 7 hours on average (mu= 7) HA: kids in a town do not sleep 7 hours on average (mu=! 7)

Your alternative hypothesis is two sided, so you need a two sided test. If you had a one sided alternative hypothesis, you could use a one sided test, but then you would not be able to look for the "other" side. E.g. if your HA is "Kids in a town sleep longer than 7 hours" then, no matter what you got for t, you would not be able to look for evidence that they slept less.

If you choose the type of test after looking at the data, you are like the famous story about the guy who fired shots at a barn and then drew targets around the holes. That's not good shooting, that's cheating.

In general, it is good to use two sided hypotheses and tests. For one thing, they let you be surprised. And, as Herman Friedman (my favorite professor in grad school) used to say "If you're not surprised, you haven't learned anything."

$\endgroup$
6
  • 2
    $\begingroup$ +1 for the quote at the end. we need more of this! $\endgroup$ Apr 16 at 12:17
  • $\begingroup$ Dear Peter, thank you so much for this explanation, and I really loved the quote too! :) $\endgroup$
    – Karen
    Apr 16 at 14:37
  • $\begingroup$ I wonder if I'm right to think that one can't switch from a two-sided to a one-sided test after viewing the data, because for the two-sided t-test you divide the significance level (alpha=0.05) between two tails: if you find p-value< 0.025, then you reject the null hypothesis. In a one-sided test you don't divide: a p-value < 0.05 rejects your null hypothesis. So, switching from two-sided to one-sided testing, you double the chance of rejecting the null hypothesis. $\endgroup$
    – Karen
    Apr 16 at 14:37
  • 1
    $\begingroup$ Yess, I see! This matter is much clearer to me now. Thank you a lot for helping me! :) $\endgroup$
    – Karen
    Apr 16 at 15:58
  • 2
    $\begingroup$ Does it matter that the hypothesis being tested makes biological or cultural sense? Presumably this is one of those frictionless statistics scenarios from a textbook rather than a real study of sleep hours which would be far, far better performed as estimation rather than a hypothesis test, or even a significance test. $\endgroup$ Apr 19 at 1:07
2
$\begingroup$

It's more complicated than you might expect, and more complicated than the current answers tell you.

The number of tails for a test has been argued back and forth many times, but as far as I can tell, the reasons to prefer two tails all relate to error rates. If you are doing a Neyman–Pearsonian hypothesis test with a dichotomised 'significant'/'not significant' result that leads to a decision to act as if the null hypothesis is false if the result is 'significant' and does not otherwise, then yes, go ahead and use two tails. However, you probably should not be doing a Neyman–Pearsonian hypothesis test! Such a test has quite limited applicability to most types of scientific study.

A common alternative to the Neyman–Pearsonian hypothesis test is a Fisherian significance test. A significance test yields a p-value that serves as an index of the strength of evidence in the data against the null hypothesis (or against the statistical model) and that can be used as an input in a thoughtful decision process. Much more useful for science.

If you do not know which type of test you are using (most people do not!) then you are likely using a hybrid. Read this chapter to learn much more: A reckless guide to p-values. There is a paragraph about tails in that chapter, but for the full argument and some references you should download this ArXived paper: To P or not to P: on the evidential nature of P-values and their place in scientific inference

The thing about tails is that if you are using the p-value as an index of the evidence in the data concerning the null hypothesis in question then the tails do not matter! A one-sided p-value of 0.002 indicates exactly the same strength of evidence as a one-sided p-value of 0.098, only with the evidence pointing to an effect in the opposite direction. And the two-tailed result of p=0.004 denotes the same evidence, but with the effect direction not encoded.

However you do it, you must say how many tails you have used.

$\endgroup$
1
$\begingroup$

Let's suppose your procedure is:

1- Look at which side points the estimate.

2- Perform a single-sided test using that side with significance alpha (for example 5%).

It should be noted that this is equivalent to performing the two one-sided tests and using the result of the one with lowest p-value.

Then, if the null hypothesis is true, what is the probability of rejecting it?

In your process, the probability of rejecting it is 2·alpha (10% in the example). Therefore, you are actually running a test with significance 2·alpha (10%), not alpha.

Of course, you could correct by performing the test in step 2 with significance alpha/2 (2,5%), but then you would be just running a two-sided test.

In summary, a two sided test is equivalent to the procedure you propose (or to making two one-sided tests) but correcting the significance level of each test to account for the repeated tests in order to keep the global probability of type I errors equal to the nominal significance.

$\endgroup$
0
$\begingroup$

Peter’s answer is very good but I’ll add that the reason that people will say it’s cheating is that if you change the “sides” you’re altering the hypothesis and also changing the size and location of the critical region of the t-distribution, thus changing alpha and the power of the test. If you do this after you see the data, the procedure is not valid since the meanings of “reject” and of p-values are predicated on first stating the hypothesis, then choosing alpha, then testing. As Peter said, it’s backward, but it’s also in violation of Neyman and Pearson’s procedure, yielding your results potentially meaningless. So this is why people often default to two sided tests, even though they actually have less power in your situation: theory suggests the effect is indeed in a particular direction but you run a two-tailed test anyway.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.