# Two-tailed tests… I'm just not convinced. What's the point?

The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site.

... consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize your ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.

After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests... I understand the basic math and increased detection ability of one tailed tests, etc... But I just can't wrap around my head around one thing... What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.

Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?

Another quote from the same page:

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" - 95%, 99%, 99.9%... Is arbitrary to begin with.

• This seems like a perfectly good question to me, +1. – gung May 23 '18 at 14:31
• While it is absolutely clear that you should design your experiment and your tests before collecting data, I find their example on drugs rather intriguing given the fact that new drugs are often tested with a 1-sided test without much outcry. – P-Gn May 23 '18 at 14:34
• @user1735003 an ironic article to come across considering many mood/behavior regulating pharmaceutical trials are coming under increasing scrutiny for observer bias. An interesting Cochrane on Ritalin here. "Claimed superiority of placebo" is what any trialist would call "harm", so I don't find it inconceivable in the least. But in these trials, if studies stop for harm, the signal is from adverse events. – AdamO May 23 '18 at 16:57
• "You have your sample mean. You have your population mean...What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?". No, the whole point of hypothesis testing is that you don't have the population mean, and you are using the sample mean to test out an assumption about the population mean(the null hypothesis). So there is no "clearly see that the sample mean is way off", because it is precisely what you're testing, not a given. – StAtS May 23 '18 at 17:14
• the trouble's that often you don't know the polarity, so you have to run the two tailed test. imagine plugging voltmeter into the DC power supply when you don't know which plug is POSITIVE – Aksakal May 23 '18 at 19:57

Think of the data as the tip of the iceberg - all you can see above the water is the tip of the iceberg but in reality you are interested in learning something about the entire iceberg.

Statisticians, data scientists and others working with data are careful to not let what they see above the water line influence and bias their assessment of what's hidden below the water line. For this reason, in a hypothesis testing situation, they tend to formulate their null and alternative hypotheses before they see the tip of the iceberg, based on their expectations (or lack thereof) of what might happen if they could view the iceberg in its entirety.

Looking at the data to formulate your hypotheses is a poor practice and should be avoided - it's like putting the cart before the horse. Recall that the data come from a single sample selected (hopefully using a random selection mechanism) from the target population/universe of interest. The sample has its own idiosyncracies, which may or may not be reflective of the underlying population. Why would you want your hypotheses to reflect a narrow slice of the population instead of the entire population?

Another way to think about this is that, every time you select a sample from your target population (using a random selection mechanism), the sample will yield different data. If you use the data (which you shouldn't!!!) to guide your specification of the null and alternative hypotheses, your hypotheses will be all over the map, essentially driven by the idiosyncratic features of each sample. Of course, in practice we only draw one sample, but it would be a very disquieting thought to know that if someone else performed the same study with a different sample of the same size, they would have to change their hypotheses to reflect the realities of their sample.

One of my graduate school professors used to have a very wise saying: "We don't care about the sample, except that it tells us something about the population". We want to formulate our hypotheses to learn something about the target population, not about the one sample we happened to select from that population.

• @subhashc.davar: Just because you don't see the relevance of my answer, it doesn't mean someone else won't. Please be mindful that answers are for the entire community not just for the person who asked the question. I'd be happy to delete my answer if you feel strongly about this. – Isabella Ghement May 23 '18 at 16:56
• @subhashc.davar An example may help: Say you test if a snack affects performance. You run the experiment and find a slight score gain in the snackers. Great! Run a one tailed test to see if snackers > non-snackers. Problem: what would you have done if you drew a sample where snackers performed worse? Would you have done a one-tailed test for snackers < non-snackers? If so, you're committing an error and letting the sample idiosyncracies guide your testing. – R.M. May 23 '18 at 17:32
• An anecdote from my professor: "We visited a friend's newborn daughter at a maternity ward. 20 kids and 18 of the 20 were wearing pink hats. So I did what any statistician would do: calculated a p-value for gender in fact being 50/50. It was very statistically significant. So who wants to write this paper with me? No one? Why? You can't use data which generated a hypothesis to test a hypothesis." – AdamO May 23 '18 at 20:02
• The horse actually is supposed to go before the cart. The common idiom is "to put the cart before the horse". – guenthmonstr May 26 '18 at 0:28
• @KennyEvitt yes exactly right. Incidental findings are important and should be reported, but they should not be sold as prespecified hypotheses. – AdamO May 29 '18 at 14:07

I think when considering your question it helps if you try to keep the goal/selling points of null-hypothesis significance testing (NHST) in mind; it's just one paradigm (albeit a very popular one) for statistical inference, and the others have their own strengths as well (e.g., see here for a discussion of NHST relative to Bayesian inference). What's the big perk of NHST?: Long-run error control. If you follow the rules of NHST (and sometimes that is a very big if), then you should have a good sense of how likely you are to be wrong with the inferences you make, in the long run.

One of the persnickety rules of NHST is that, without further alteration to your testing procedure, you only get to take one look at your test of interest. Researchers in practice often ignore (or are not aware of) this rule (see Simmons et al., 2012), conducting multiple tests after adding waves of data, checking their $p$-values after adding/removing variables to their models, etc. The problem with this is that researchers are rarely neutral with respect to outcome of NHST; they are keenly aware that significant results are more likely to be published than are non-significant results (for reasons that are both misguided and legitimate; Rosenthal, 1979). Researchers are therefore often motivated to add data/amend models/select outliers and repeatedly test until they "uncover" a significant effect (see John et al., 2011, a good introduction).

A counterintuitive problem is created by the above practices, described nicely in Dienes (2008): if researchers will keep adjusting their sample/design/models until significance is achieved, then their desired long-run error rates of false-positive findings (often $\alpha =.05$) and false-negative findings (often $\beta =.20$) will each approach 1.0 and 0.0, respectively (i.e., you will always reject $H_0$, both when it's false and when it's true).

In the context of your specific questions, researchers use two-tailed tests as a default when they don't want to make particular predictions with respect to the direction of the effect. If they are wrong in their guess, and run a one-tailed test in the direction of the effect, their long-run $\alpha$ will be inflated. If they look at descriptive statistics and run a one-tailed test based on their eyeballing of the trend, their long-run $\alpha$ will be inflated. You might think this isn't a huge problem, in practice, that the $p$-values lose their long-run meaning, but if they don't retain their meaning, it begs the question of why you are using an approach to inference that prioritizes long-run error control.

Lastly (and as a matter of personal preference), I would have less of a problem if you first conducted a two-tailed test, found it non-significant, then did the one-tailed test in the direction the first test implied, and found it to be significant if (and only if) you performed a strict confirmatory replication of that effect in another sample, and published the replication in the same paper. Exploratory data analysis--with error-rate inflating flexible analysis practice--is fine, as long as you are able to replicate your effect in a new sample without that same analytic flexibility.

References

Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. Palgrave Macmillan.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-532.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological bulletin, 86(3), 638.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.

• Very nice answer. Helped me pull together some concerns I had during my recent delve into research papers (as a layman), seemingly confirming the idea that one-tailed p-values can only be "trusted" if you have existing reason to be confident in the "negative correlation" direction being false. – Venryx Dec 25 '18 at 22:51

Unfortunately, the motivating example of drug development is not a good one as it's not what we do to develop drugs. We use different, more stringent rules to stop the study if trends are on the side of harm. This is for the safety of the patients and also because the drug is unlikely to magically swing in the direction of a meaningful benefit.

So why do two tailed tests? (when in most cases we have some a priori notion of the possible direction of effect we're trying to model)

The null hypothesis should bear some resemblance to belief in the sense of being plausible, informed, and justified. In most cases, people agree an "uninteresting result" is when there is 0 effect, whereas a negative or a positive effect is of equal interest. It is very hard to articulate a composite null hypothesis, e.g. the case where we know the statistic could be equal to or less than a certain amount. One must be very explicit about a null hypothesis to make sense of their scientific findings. It's worth pointing out that the manner in which one conducts a composite hypothesis test is that the statistic under the null hypothesis assumes the most consistent value within the range of the observed data. So if the effect is in the positive direction as expected, the null value is taken to be 0 anyway, and we've mooted needlessly.

A two tailed test amounts to conducting two one-sided tests with control for multiple comparisons! The two tailed test is actually partly valued because it ends up being more conservative in the long run. When we have good belief about the direction of effect, the two tailed tests will yield false positives half as often with very little overall effect on power.

In the case of evaluating a treatment in a randomized controlled trial, if you tried to sell me a one-sided test, I would stop you to ask, "Well wait, why would we believe the treatment is actually harmful? Is there actually evidence to support this? Is there even equipoise [an ability to demonstrate a beneficial effect]?" The logical inconsistency behind the one-sided test calls the whole research into question. If truly nothing is known, any value other than 0 is considered interesting and the two tailed test is not just a good idea, it's necessary.

One way to approach it is to temporarily forget about hypothesis testing and think about confidence intervals instead. One-sided tests correspond to one-sided confidence intervals and two-sided tests correspond to two-sided confidence intervals.

Suppose that you want to estimate the mean of a population. Naturally, you take a sample and compute a sample mean. There is no reason to take a point-estimate at face value, so you express your answer in terms of an interval that you are reasonably confident contains the true mean. What type of interval do you choose? A two-sided interval is by far the more natural choice. A one-sided interval only makes sense when you simply don't care about finding either an upper bound or a lower bound of your estimate (because you believe that you already know a useful bound in one direction). How often are you really that sure about the situation?

Perhaps switching the question to confidence intervals doesn't really nail it down, but it is methodologically inconsistent to prefer one-tailed tests but two-sided confidence intervals.

After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests... I understand the basic math and increased detection ability of one tailed tests, etc... But I just can't wrap around my head around one thing... What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.

The problem is that you don't know the population mean. I have never encountered a real world scenario that I know the true population mean.

Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?

I read your paragraph several times, but I'm still not sure about your arguments. Do you want to rephrase it? You fail to "test" if your data doesn't land you in your chosen critical regions.

I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

The quote is correct because hacking a p-value is inappropriate. How much do we know about p-hacking "in the wild"? has more details.

Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" - 95%, 99%, 99.9%... Is arbitrary to begin with. Help?

It is arbitrary. That's why data scientists generally report the magnitude of the p-value itself (not just significant or insignificant), and also the effects size.

• To be clear, I am not trying to challenge the very foundations of statistical inference. As I stated, I have only just learned the very basics and am having trouble understanding how any potential findings could be missed by failing to use the correct test. – FromTheAshes May 23 '18 at 11:09
• Say your buddy, Joe, invents a new product that he claims greatly enhances plant growth. Intrigued, you devise a robust study with a control group, and treatment group. Your null hyp. is that there will be no change in growth, your alternative hyp. is that Joe's magic spray significantly increases growth - so a one-tailed test. 2 weeks later, you make your final observations and analyse the results. The mean growth of the treatment group turns out to be over 5 standard errors BELOW the control's. How is this very significant finding any less obvious or valid because of your choice of test? – FromTheAshes May 23 '18 at 11:28
• If I ask you to call heads or tails for a coin toss, the probability of your predicting the outcome is 50/50 (assuming a balanced coin and an honest flipper). However, if I flip the coin first and let you look at it and then make your prediction, it will no longer be 50/50. If you are conducting a one-tailed test with an alpha level of .01 but then flip the direction of the test after seeing the results because p<.01 in the other direction, your risk of a Type I error is no long .01 but much higher. Note that the observed p-value and Type I error rate are not the same thing. – dbwilson May 23 '18 at 12:18
• @FromTheAshes there is nothing wrong with trying to challenge the very foundations. Statistical hypothesis testing is not useless, but it does contain massive logical flaws, and it is absolutely reasonable to challenge them! – Flounderer May 24 '18 at 15:13

Well, all difference relies in the question you want to answer. If the question is: "Is one group of values bigger than the other?" you can use a one tailed test. To answer the question: "Are these groups of values different?" you use the two tailed test. Take into consideration that a set of data may be statistically higher than another, but not statistically different... and that's statistics.

• 'If the question is: "Is one group of values bigger than the other?" you can use a one tailed test.' More precisely, if the question is "Is *this particular group bigger than the others", then you should use a two-tailed test. – Acccumulation May 23 '18 at 21:39
• It should be noted that it's kind of implied that if you're asking that question that "And by the way if it looks like the other group is actually bigger then I don't care". If you would see the opposite of what you would expect and then go on to flip the direction of the hypothesis test then you were just lying to yourself all along and should have done a two-tailed test to begin with. – Dason May 24 '18 at 21:17

But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

The alpha value is the probability that you will reject the null, given that the null is true. Suppose your null is that the sample mean is normally distributed with mean zero. If P(sample mean>1|H0) = .05, then the rule "Collect a sample, and reject the null if the sample mean is greater than 1" has a probability, given that the null is true, of 5% of rejecting the null. The rule "Collect a sample, and if the sample mean is positive, then reject the null if the sample mean is greater than 1, and if the sample mean is negative, reject the null if the sample mean is less than 1" has a probability, given that the null is true, of 10% of rejecting the null. So the first rule has an alpha of 5%, and the second rule has an alpha of 10%. If you start out with a two-tailed test, and then change it to a one-tailed test based on the data, then you're following the second rule, so it would be inaccurate to report your alpha as 5%. The alpha value depends not only on what the data is, but what rules you are following in analyzing it. If you're asking why use a metric that has this property, rather than something that depends only on the data, that is a more complicated question.

This is just one arbitrary way to look at it: What is a statistical test used for? Probably the most frequent reason to perform a test is because you want to convince people (i. e. editors, reviewers, readers, audience) that your results are "far enough off random" to be noteworthy. And somehow we concluded that $p < \alpha = 0.05$ is the arbitrary, yet universal truth.

For any other sensible reason to perform tests, you would never settle for a fixed $\alpha$ of $0.05$, but you would vary your $\alpha$ from case to case, depending on how important the consequences were, that you draw from the test.

Back to convincing people, that something is "far enough from just random" to meet a universal criterion of noteworthiness. We have an insensible, yet universally accepted, criterion, that we believe thinks to be "not random" at $\alpha=0.05$ for two-sided testing. An equivalent criterion would be to look at the data, decide which way to test and draw the line at $\alpha=0.025$. The second one is equivalent to the first one, but it is not what we have historically settled with.

Once you start to do one-sided tests with $\alpha=0.05$ you make yourself suspicious of undue behaviour, of fishing for significance. Don't do that, if you want to convince people!

Then, of course, there is this thing called researchers degree of freedom. You can find significance in any kind of data, if you have sufficient data and are free to test it in as many ways as you wish. This is why you are meant to decide on the test you conduct before looking at the data. Everything else leads to irreproducible test results. I advise to go to youtube and look at Andrew Gelmans talk "Crimes on data for more on that.

• Hmm, the null hypothesis is not that results are random. This would be confusing to clinicians and scientists who very much see the results of their work as achieving a fixed outcome. – AdamO May 23 '18 at 16:31
• Your "Once you start to do one-sided tests with $α=0.05$ ..." point is important. The reason $0.05$ is so common is that RA Fisher's practical experience at Rothamsted was that being more than $2$ standard deviations from the expected value was generally worth further investigation, and from this he chose a two-tailed $5\%$ test as his rule of thumb, not the other way round. Thus the one-tailed equivalent would be $2.5\%$ – Henry May 26 '18 at 18:02

At first glance, neither of these statements make the assertion that a two-sided test is 'superior' to a one-sided study. There simply needs to be a logical connection from the research hypothesis being tested linked to the statistical inference being tested.

For instance:

... consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.

First off this is a drug study. So being incorrect in the opposite direction has social significance beyond the framework of statistics. So like many have said health isn't the best to make generalizations.

In the quote above, it seems to be about testing a drug when another already exists. So to me, this implies your drug is assumed to be already effective. The statement is in regard to the comparison of two effective drugs thereafter. When comparing these distributions if you're neglecting one side of the population for the sake of improving its comparative results? It’s not only a biased conclusion but the comparison is no longer a valid one to justify: you’re comparing apples to oranges.

Similarly, there very well may be point estimates that for the sake of statistical inference made no difference to the conclusion, but are very much of social importance. That's because our sample represents people's lives: something that cannot "re-occur" and is invaluable.

Alternatively, the statement implies the researcher has an incentive: "you wish to maximize your ability to detect the improvement..." This notion is non-trivial to the case being isolated as a bad protocol.

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

Again here it implies the researcher is 'switching' his test: from a two-sided to a one-sided. This is never appropriate. It's imperative to have a research purpose before testing. By always defaulting to the convenience of a two-sided approach -- researchers conveniently fail to more rigorously understand the phenomenon.

Here's a paper on this very topic, in fact, making the case that two-sided tests have been overused.

It blames the over-use of a two-sided test on the lack of a:

clear distinction and a logical linkage between the research hypothesis and its statistical hypothesis

It takes the position and stance that researchers:

may not be aware of the difference between the two expressive modes or aware of the logical flow in which the research hypothesis should be translated into the statistical hypothesis. A convenience-oriented mixing of the research and statistical hypotheses may be a cause of the overuse of two-tailed testing even in situations where the use of two-tailed testing is inappropriate.

what is needed is to grasp the exact statistics in interpreting statistical testing results. Being inexact under the name of being conservative is not recommendable. In that sense, the authors think that merely reporting testing results such as “It was found to be statistically significant at the 0.05 level of significance (i.e., p < 0.05).” is not good enough.

Although two-tailed testing is more conservative in theory, it decouples the link between the directional research hypothesis and its statistical hypothesis, possibly leading to doubly inflated p values.

The authors have also shown that the argument for finding the significant result in the opposite direction has meaning only in the context of discovery rather than in the context of
justification. In the case of testing the research hypothesis and its underlying theory, researchers should not simultaneously address the context of discovery and that of justification.

https://www.sciencedirect.com/science/article/pii/S0148296312000550

Regarding the 2nd point

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

we have that, if the null is true, the first, two-tailed, test falsely rejects with probability $\alpha$, but the one-sided may also reject in the second stage.

The overall rejection probability will hence exceed $\alpha$, and you are not testing at the level you believe to be testing anymore - you more often get false rejections than in $\alpha\cdot 100\%$ of the cases in which the strategy is applied to true null hypotheses.

Overall, we seek $$P(\text{two-sided rejects or one-sided does, but two sided doesn't})$$ which we may express as $$P(\text{two-sided rejects} \cup \text{(one-sided does} \cap \text{two sided doesn't)})$$ The two events in the union are disjoint, so that we are after $$P(\text{two-sided rejects}) +P(\text{one-sided does} \cap \text{two sided doesn't})$$ For the second term, there is probability mass $\alpha/2$ between the upper $1-\alpha$ and $1-\alpha/2$ quantiles (i.e., the rejection points of the one-sided and two-sided tests), which is the joint probability of the two-sided test not rejecting but the one-sided doing so. Hence, $$P(\text{one-sided does} \cap \text{two sided doesn't})=\alpha/2$$ so that the overall rejection probability of this strategy is $$\alpha+\frac{\alpha}{2}>\alpha$$ Effectively, we just add up the probabilities that the test statistic lands to the left of the $\alpha/2$-quantile, between the upper $1-\alpha$ and $1-\alpha/2$ quantiles or to the right of the $1-\alpha/2$-quantile.

Here is a little numerical illustration:

n <- 100
alpha <- 0.05

two.sided <- function (x, alpha=0.05) (sqrt(n)*abs(mean(x)) > qnorm(1-alpha/2)) # returns one if two-sided test rejects, 0 else
one.sided <- function (x, alpha=0.05) (sqrt(n)*mean(x) > qnorm(1-alpha))        # returns one if one-sided test rejects, 0 else

reps <- 1e8

two.step <- rep(NA,reps)
for (i in 1:reps){
x <- rnorm(n) # generate data from a N(0,1) distribution, so that the test statistic sqrt(n)*mean(x) is also N(0,1) under H_0: mu=0
two.step[i] <- ifelse(two.sided(x)==0, one.sided(x), 1) # first conducts two-sided test, then one-sided if two-sided fails to reject
}
> mean(two.step)
[1] 0.07505351

Often a significance test is performed for the null hypothesis against an alternative hypothesis. This is when one-tailed versus two-tailed make a difference.

• For p-values this (two or one sided) does not matter! The point is that you select a criterium that only occurs a fraction $\alpha$ of the time when the null hypothesis is true. This is either two small pieces of both tails, or one big piece of one tail, or something else.

Type I error rate is not different for one or two sided tests.

• On the other hand, for the power it matters.

If your alternative hypothesis is asymmetric, then you'd wish to focus the criterium to reject the null hypothesis only on this tail/end; such that when the alternative hypothesis is true then you are less likely to not reject ("accept") the null hypothesis.

If your alternative hypothesis is symmetric (you don't care to place more or less power on one specific side), and deflection/effect on both sides is equally expected (or just unknown/uninformed), then it is more powerful to use a two-sided test (you are not loosing 50% power for the tail that you are not testing and where you will make many type II errors).

Type II error rate is different for one and two sided tests and depending on the alternative hypothesis as well.

It is becoming more a bit like a Bayesian concept now when we start involving preconceptions about whether or not we expect an effect to fall on one side or on both sides, and when we wish to use a test (to see if we can falsify a null-hypothesis) to 'confirm' or make more probable something like an effect.