Two-tailed tests... I'm just not convinced. What's the point?

Question

The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site.

... consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize your ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.

After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests... I understand the basic math and increased detection ability of one tailed tests, etc... But I just can't wrap around my head around one thing... What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.

Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?

Another quote from the same page:

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" - 95%, 99%, 99.9%... Is arbitrary to begin with.

Answer 1

Think of the data as the tip of the iceberg – all you can see above the water is the tip of the iceberg but in reality you are interested in learning something about the entire iceberg.

Statisticians, data scientists and others working with data are careful to not let what they see above the water line influence and bias their assessment of what's hidden below the water line. For this reason, in a hypothesis testing situation, they tend to formulate their null and alternative hypotheses before they see the tip of the iceberg, based on their expectations (or lack thereof) of what might happen if they could view the iceberg in its entirety.

Looking at the data to formulate your hypotheses is a poor practice and should be avoided – it's like putting the cart before the horse. Recall that the data come from a single sample selected (hopefully using a random selection mechanism) from the target population/universe of interest. The sample has its own idiosyncracies, which may or may not be reflective of the underlying population. Why would you want your hypotheses to reflect a narrow slice of the population instead of the entire population?

Another way to think about this is that, every time you select a sample from your target population (using a random selection mechanism), the sample will yield different data. If you use the data (which you shouldn't!!!) to guide your specification of the null and alternative hypotheses, your hypotheses will be all over the map, essentially driven by the idiosyncratic features of each sample. Of course, in practice we only draw one sample, but it would be a very disquieting thought to know that if someone else performed the same study with a different sample of the same size, they would have to change their hypotheses to reflect the realities of their sample.

One of my graduate school professors used to have a very wise saying: "We don't care about the sample, except that it tells us something about the population". We want to formulate our hypotheses to learn something about the target population, not about the one sample we happened to select from that population.

Answer 2

I think when considering your question it helps if you try to keep the goal/selling points of null-hypothesis significance testing (NHST) in mind; it's just one paradigm (albeit a very popular one) for statistical inference, and the others have their own strengths as well (e.g., see here for a discussion of NHST relative to Bayesian inference). What's the big perk of NHST?: Long-run error control. If you follow the rules of NHST (and sometimes that is a very big if), then you should have a good sense of how likely you are to be wrong with the inferences you make, in the long run.

One of the persnickety rules of NHST is that, without further alteration to your testing procedure, you only get to take one look at your test of interest. Researchers in practice often ignore (or are not aware of) this rule (see Simmons et al., 2012), conducting multiple tests after adding waves of data, checking their $p$-values after adding/removing variables to their models, etc. The problem with this is that researchers are rarely neutral with respect to outcome of NHST; they are keenly aware that significant results are more likely to be published than are non-significant results (for reasons that are both misguided and legitimate; Rosenthal, 1979). Researchers are therefore often motivated to add data/amend models/select outliers and repeatedly test until they "uncover" a significant effect (see John et al., 2011, a good introduction).

A counterintuitive problem is created by the above practices, described nicely in Dienes (2008): if researchers will keep adjusting their sample/design/models until significance is achieved, then their desired long-run error rates of false-positive findings (often $\alpha =.05$) and false-negative findings (often $\beta =.20$) will each approach 1.0 and 0.0, respectively (i.e., you will always reject $H_0$, both when it's false and when it's true).

In the context of your specific questions, researchers use two-tailed tests as a default when they don't want to make particular predictions with respect to the direction of the effect. If they are wrong in their guess, and run a one-tailed test in the direction of the effect, their long-run $\alpha$ will be inflated. If they look at descriptive statistics and run a one-tailed test based on their eyeballing of the trend, their long-run $\alpha$ will be inflated. You might think this isn't a huge problem, in practice, that the $p$-values lose their long-run meaning, but if they don't retain their meaning, it begs the question of why you are using an approach to inference that prioritizes long-run error control.

Lastly (and as a matter of personal preference), I would have less of a problem if you first conducted a two-tailed test, found it non-significant, then did the one-tailed test in the direction the first test implied, and found it to be significant if (and only if) you performed a strict confirmatory replication of that effect in another sample, and published the replication in the same paper. Exploratory data analysis--with error-rate inflating flexible analysis practice--is fine, as long as you are able to replicate your effect in a new sample without that same analytic flexibility.

References

Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. Palgrave Macmillan.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-532.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological bulletin, 86(3), 638.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.

Answer 3

Unfortunately, the motivating example of drug development is not a good one as it's not what we do to develop drugs. We use different, more stringent rules to stop the study if trends are on the side of harm. This is for the safety of the patients and also because the drug is unlikely to magically swing in the direction of a meaningful benefit.

So why do two tailed tests? (when in most cases we have some a priori notion of the possible direction of effect we're trying to model)

The null hypothesis should bear some resemblance to belief in the sense of being plausible, informed, and justified. In most cases, people agree an "uninteresting result" is when there is 0 effect, whereas a negative or a positive effect is of equal interest. It is very hard to articulate a composite null hypothesis, e.g. the case where we know the statistic could be equal to or less than a certain amount. One must be very explicit about a null hypothesis to make sense of their scientific findings. It's worth pointing out that the manner in which one conducts a composite hypothesis test is that the statistic under the null hypothesis assumes the most consistent value within the range of the observed data. So if the effect is in the positive direction as expected, the null value is taken to be 0 anyway, and we've mooted needlessly.

A two tailed test amounts to conducting two one-sided tests with control for multiple comparisons! The two tailed test is actually partly valued because it ends up being more conservative in the long run. When we have good belief about the direction of effect, the two tailed tests will yield false positives half as often with very little overall effect on power.

In the case of evaluating a treatment in a randomized controlled trial, if you tried to sell me a one-sided test, I would stop you to ask, "Well wait, why would we believe the treatment is actually harmful? Is there actually evidence to support this? Is there even equipoise [an ability to demonstrate a beneficial effect]?" The logical inconsistency behind the one-sided test calls the whole research into question. If truly nothing is known, any value other than 0 is considered interesting and the two tailed test is not just a good idea, it's necessary.

Answer 4

One way to approach it is to temporarily forget about hypothesis testing and think about confidence intervals instead. One-sided tests correspond to one-sided confidence intervals and two-sided tests correspond to two-sided confidence intervals.

Suppose that you want to estimate the mean of a population. Naturally, you take a sample and compute a sample mean. There is no reason to take a point-estimate at face value, so you express your answer in terms of an interval that you are reasonably confident contains the true mean. What type of interval do you choose? A two-sided interval is by far the more natural choice. A one-sided interval only makes sense when you simply don't care about finding either an upper bound or a lower bound of your estimate (because you believe that you already know a useful bound in one direction). How often are you really that sure about the situation?

Perhaps switching the question to confidence intervals doesn't really nail it down, but it is methodologically inconsistent to prefer one-tailed tests but two-sided confidence intervals.

Answer 5

After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests... I understand the basic math and increased detection ability of one tailed tests, etc... But I just can't wrap around my head around one thing... What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.

The problem is that you don't know the population mean. I have never encountered a real world scenario that I know the true population mean.

Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?

I read your paragraph several times, but I'm still not sure about your arguments. Do you want to rephrase it? You fail to "test" if your data doesn't land you in your chosen critical regions.

I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

The quote is correct because hacking a p-value is inappropriate. How much do we know about p-hacking "in the wild"? has more details.

Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" - 95%, 99%, 99.9%... Is arbitrary to begin with. Help?

It is arbitrary. That's why data scientists generally report the magnitude of the p-value itself (not just significant or insignificant), and also the effects size.

Answer 6

Well, all difference relies in the question you want to answer. If the question is: "Is one group of values bigger than the other?" you can use a one tailed test. To answer the question: "Are these groups of values different?" you use the two tailed test. Take into consideration that a set of data may be statistically higher than another, but not statistically different... and that's statistics.

Answer 7

But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

The alpha value is the probability that you will reject the null, given that the null is true. Suppose your null is that the sample mean is normally distributed with mean zero. If P(sample mean>1|H0) = .05, then the rule "Collect a sample, and reject the null if the sample mean is greater than 1" has a probability, given that the null is true, of 5% of rejecting the null. The rule "Collect a sample, and if the sample mean is positive, then reject the null if the sample mean is greater than 1, and if the sample mean is negative, reject the null if the sample mean is less than 1" has a probability, given that the null is true, of 10% of rejecting the null. So the first rule has an alpha of 5%, and the second rule has an alpha of 10%. If you start out with a two-tailed test, and then change it to a one-tailed test based on the data, then you're following the second rule, so it would be inaccurate to report your alpha as 5%. The alpha value depends not only on what the data is, but what rules you are following in analyzing it. If you're asking why use a metric that has this property, rather than something that depends only on the data, that is a more complicated question.

Answer 8

Regarding the 2nd point

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

we have that, if the null is true, the first, two-tailed, test falsely rejects with probability $\alpha$, but the one-sided may also reject in the second stage.

The overall rejection probability will hence exceed $\alpha$, and you are not testing at the level you believe to be testing anymore - you more often get false rejections than in $\alpha\cdot 100\%$ of the cases in which the strategy is applied to true null hypotheses.

Overall, we seek $$ P(\text{two-sided rejects or one-sided does, but two sided doesn't}) $$ which we may express as $$ P(\text{two-sided rejects} \cup \text{(one-sided does} \cap \text{two sided doesn't)}) $$ The two events in the union are disjoint, so that we are after $$ P(\text{two-sided rejects}) +P(\text{one-sided does} \cap \text{two sided doesn't}) $$ For the second term, there is probability mass $\alpha/2$ between the upper $1-\alpha$ and $1-\alpha/2$ quantiles (i.e., the rejection points of the one-sided and two-sided tests), which is the joint probability of the two-sided test not rejecting but the one-sided doing so. Hence, $$P(\text{one-sided does} \cap \text{two sided doesn't})=\alpha/2 $$ so that the overall rejection probability of this strategy is $$\alpha+\frac{\alpha}{2}>\alpha$$ Effectively, we just add up the probabilities that the test statistic lands to the left of the $\alpha/2$-quantile, between the upper $1-\alpha$ and $1-\alpha/2$ quantiles or to the right of the $1-\alpha/2$-quantile.

Here is a little numerical illustration:

n <- 100
alpha <- 0.05

two.sided <- function (x, alpha=0.05) (sqrt(n)*abs(mean(x)) > qnorm(1-alpha/2)) # returns one if two-sided test rejects, 0 else
one.sided <- function (x, alpha=0.05) (sqrt(n)*mean(x) > qnorm(1-alpha))        # returns one if one-sided test rejects, 0 else

reps <- 1e8

two.step <- rep(NA,reps)
for (i in 1:reps){
  x <- rnorm(n) # generate data from a N(0,1) distribution, so that the test statistic sqrt(n)*mean(x) is also N(0,1) under H_0: mu=0
  two.step[i] <- ifelse(two.sided(x)==0, one.sided(x), 1) # first conducts two-sided test, then one-sided if two-sided fails to reject
}
> mean(two.step)
[1] 0.07505351

Answer 9

This is just one arbitrary way to look at it: What is a statistical test used for? Probably the most frequent reason to perform a test is because you want to convince people (i. e. editors, reviewers, readers, audience) that your results are "far enough off random" to be noteworthy. And somehow we concluded that $p < \alpha = 0.05$ is the arbitrary, yet universal truth.

For any other sensible reason to perform tests, you would never settle for a fixed $\alpha$ of $0.05$, but you would vary your $\alpha$ from case to case, depending on how important the consequences were, that you draw from the test.

Back to convincing people, that something is "far enough from just random" to meet a universal criterion of noteworthiness. We have an insensible, yet universally accepted, criterion, that we believe thinks to be "not random" at $\alpha=0.05$ for two-sided testing. An equivalent criterion would be to look at the data, decide which way to test and draw the line at $\alpha=0.025$. The second one is equivalent to the first one, but it is not what we have historically settled with.

Once you start to do one-sided tests with $\alpha=0.05$ you make yourself suspicious of undue behaviour, of fishing for significance. Don't do that, if you want to convince people!

---

Then, of course, there is this thing called researchers degree of freedom. You can find significance in any kind of data, if you have sufficient data and are free to test it in as many ways as you wish. This is why you are meant to decide on the test you conduct before looking at the data. Everything else leads to irreproducible test results. I advise to go to youtube and look at Andrew Gelmans talk "Crimes on data for more on that.

Answer 10

At first glance, neither of these statements make the assertion that a two-sided test is 'superior' to a one-sided study. There simply needs to be a logical connection from the research hypothesis being tested linked to the statistical inference being tested.

For instance:

... consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.

First off this is a drug study. So being incorrect in the opposite direction has social significance beyond the framework of statistics. So like many have said health isn't the best to make generalizations.

In the quote above, it seems to be about testing a drug when another already exists. So to me, this implies your drug is assumed to be already effective. The statement is in regard to the comparison of two effective drugs thereafter. When comparing these distributions if you're neglecting one side of the population for the sake of improving its comparative results? It’s not only a biased conclusion but the comparison is no longer a valid one to justify: you’re comparing apples to oranges.

Similarly, there very well may be point estimates that for the sake of statistical inference made no difference to the conclusion, but are very much of social importance. That's because our sample represents people's lives: something that cannot "re-occur" and is invaluable.

Alternatively, the statement implies the researcher has an incentive: "you wish to maximize your ability to detect the improvement..." This notion is non-trivial to the case being isolated as a bad protocol.

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

Again here it implies the researcher is 'switching' his test: from a two-sided to a one-sided. This is never appropriate. It's imperative to have a research purpose before testing. By always defaulting to the convenience of a two-sided approach -- researchers conveniently fail to more rigorously understand the phenomenon.

Here's a paper on this very topic, in fact, making the case that two-sided tests have been overused.

It blames the over-use of a two-sided test on the lack of a:

clear distinction and a logical linkage between the research hypothesis and its statistical hypothesis

It takes the position and stance that researchers:

may not be aware of the difference between the two expressive modes or aware of the logical flow in which the research hypothesis should be translated into the statistical hypothesis. A convenience-oriented mixing of the research and statistical hypotheses may be a cause of the overuse of two-tailed testing even in situations where the use of two-tailed testing is inappropriate.

what is needed is to grasp the exact statistics in interpreting statistical testing results. Being inexact under the name of being conservative is not recommendable. In that sense, the authors think that merely reporting testing results such as “It was found to be statistically significant at the 0.05 level of significance (i.e., p < 0.05).” is not good enough.

Although two-tailed testing is more conservative in theory, it decouples the link between the directional research hypothesis and its statistical hypothesis, possibly leading to doubly inflated p values.

The authors have also shown that the argument for finding the significant result in the opposite direction has meaning only in the context of discovery rather than in the context of
justification. In the case of testing the research hypothesis and its underlying theory, researchers should not simultaneously address the context of discovery and that of justification.

https://www.sciencedirect.com/science/article/pii/S0148296312000550

Answer 11

Often a significance test is performed for the null hypothesis against an alternative hypothesis. This is when one-tailed versus two-tailed make a difference.

---

• For p-values this (two or one sided) does not matter! The point is that you select a criterium that only occurs a fraction $\alpha$ of the time when the null hypothesis is true. This is either two small pieces of both tails, or one big piece of one tail, or something else.

  Type I error rate is not different for one or two sided tests.

• On the other hand, for the power it matters.

  If your alternative hypothesis is asymmetric, then you'd wish to focus the criterium to reject the null hypothesis only on this tail/end; such that when the alternative hypothesis is true then you are less likely to not reject ("accept") the null hypothesis.

  If your alternative hypothesis is symmetric (you don't care to place more or less power on one specific side), and deflection/effect on both sides is equally expected (or just unknown/uninformed), then it is more powerful to use a two-sided test (you are not loosing 50% power for the tail that you are not testing and where you will make many type II errors).

  Type II error rate is different for one and two sided tests and depending on the alternative hypothesis as well.

It is becoming more a bit like a Bayesian concept now when we start involving preconceptions about whether or not we expect an effect to fall on one side or on both sides, and when we wish to use a test (to see if we can falsify a null-hypothesis) to 'confirm' or make more probable something like an effect.

Answer 12

So one more answer attempt:

I guess whether to take one-tailed or two-tailed depends completely on the Alternative hypothesis.

Consider the following example of testing mean in a t-test:

$H_0: \mu=0$

$H_a: \mu \neq 0$

Now if you observe a very negative sample mean or a very positive sample mean, your hypothesis is unlikely to be true.

On the other hand, you will be willing to accept your hypothesis if your sample mean is close to $0$ whether negative or positive. Now you need to choose the interval in which, if your sample mean would fall, you wouldn't reject your null hypothesis. Obviously you'd choose an interval that has both negative and positive sides around $0$. So you choose the two side test.

But what if you don't want to test $\mu=0$, but rather $\mu\geq 0$. Now intuitively what we want to do here is that if value of sample mean comes very negative, then we can definitely reject our null. So we would want to reject null only for far negative values of sample mean.

But wait! If that's my null hypothesis how would I set my null distribution. The null distribution of the sample mean is known for some assumed value of the population parameter (here $0$). But under current null it can take many values.

Let's say we can do infinite null hypotheses. Each for assuming a positive value of $\mu$. But think of this: In our first hypothesis of $H_0: \mu=0$, if we only reject null on obsering very far negative sample mean, then every next hypothesis with $H_0: \mu>0$ would also reject it. Because for them, the sample mean is even farther from population parameter. So basically all we need to do really is just do one hypothesis but one-tailed.

So your solution becomes:

$H_0: \mu=0$

$H_a: \mu <0$

Best example is Dickey-Fuller test for stationarity.

Hope this helps. (Wanted to include diagrams but replying from mobile).

Answer 13

It's easy to see where the confusions comes from if you liberate yourself from a single dimension while remembering that a particular hypothesis you're testing is of significance of difference from 0. Whether your estimate $x$ is different from zero or not is your question here.

Ask yourself what would be the test if your variable wasn't a scalar but a vector? Imagine that you are looking at multidimensional variable $(x_1,x_2,\dots,x_n)$, i.e. a vector $\mathbf x$. You want to know whether it is far from origin or not. How would you proceed?

I have no doubt that some sort of a norm, such as Euclidian, would be your first thought: $r=||\mathbf x||$. Next, you'd want to assess whether it is so far from the origin that you have little doubt it is not at origin, i.e. how far it is from the origin or whether $r=0$?

Now, let's get back to one dimension, and see what's a norm: $r=|x|$, it's just an absolute value. Hence, for a symmetrical distribution such as Gaussian (normal) you'll end up considering quantities such as $\alpha/2$ significance.