What is a two-sided type 1 error rate?

This is probably a simple question but I was watching an online video about a scientific study. For the study, they mentioned that they used a two-tailed Type 1 error rate of 0.05. I know that a Type 1 error is the probability of rejecting the null hypothesis when it is actually true. However, I am not sure what is meant by a two-tailed Type 1 error rate. Does the 'two-tailed' refer to the fact that the hypothesis test used is two-sided? Any insights are appreciated.

• Yes, it means that the test is two-tailed. The particular phrasing has to do with allocating $0.05$ probability between the two tales, rather than putting all of that in just one tail.
– Dave
Oct 10, 2021 at 4:29

Examples of two-sided and one-sided t tests

Suppose you have data as follows: I used R to take a sample of size $$n = 30$$ from a normal distribution population with mean $$\mu = 100.$$ However, I happened to get a sample with mean $$\bar X - 95.55,$$ somewhat smaller than $$\mu = 100.$$

set.seed(1234)
x = rnorm(30, 100, 15)
summary(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
64.81   86.85   92.49   95.55  103.62  136.24

I will do a t test of the null hypothesis $$H_O: \mu =100$$ against the two-sided alternative $$H_a:\mu \ne 100.$$ I want to test at the 5% level of significance.

t.test(x, mu = 100)

One Sample t-test

data:  x
t = -1.798, df = 29, p-value = 0.08259
alternative hypothesis:
true mean is not equal to 100
95 percent confidence interval:
90.49593 100.61132
sample estimates:
mean of x
95.55363

The P-value is $$0.08259 > 0.05 = 5\%,$$ so I do not reject at the 5% level. Even though I happened to get a sample that is a bit strange, its sample mean 95.55 is not enough different from $$\mu_0 = 100$$ to reject the null hypothesis.

The P-value is the probability under the density curve of Student's t distribution with 29 degrees of freedom of getting a T statistic farther from $$0$$ than the observed $$-1.798.$$ In the figure below that amounts to the sum of the areas in the two tails outside the vertical red lines.

However, if I decide (perhaps after seeing the small value of $$\bar X)$$ to do a left-sided test of $$H_0:\mu=100$$ against $$H_a:\mu < 100,$$ also at the 5% level. Then I get P-value $$0.0413 < 0.05 = 5\%,$$ so I reject the null hypothesis. In the figure, this is the area in the left tail (only) to the left of the solid red line. (The P-value of the one-tailed test is half of the P-value of the two-sided test.)

t.test(x, mu=100, alte="less")

One Sample t-test

data:  x
t = -1.798, df = 29, p-value = 0.0413
alternative hypothesis:
true mean is less than 100
95 percent confidence interval:
-Inf 99.75543
sample estimates:
mean of x
95.55363

Even though both the two-sided and the one-sided test were at the same 5% level, we failed to reject $$H_0$$ for the two-sided test and rejected for the one-tailed test. This is because the criteria for rejection are different for the two tests. So, in a practical application, it is important to decide from the start (preferably before data are available) whether you need to do a two-sided or a one-sided test.

Notes: (1) In case you're familiar with R and want the R code for the figure, it is shown below.

hdr = "Density of T(29)"
curve(dt(x, 20), -3.5, 3.5, lwd=2,
ylab="PDF", xlab="t", main=hdr)
abline(h=0, col="green2")
abline(v=0, col="green2")
abline(v=-1.798, col="red")
abline(v=1.798, col="red", lty="dotted")

In case you want to see how P-values can be obtained in R, here they are:

1 - diff(pt(c(-1.798, 1.798), 29))
[1] 0.08259619  # 2-sided P-value
pt(-1.798, 29)
[1] 0.04129809