This is probably a simple question but I was watching an online video about a scientific study. For the study, they mentioned that they used a two-tailed Type 1 error rate of 0.05. I know that a Type 1 error is the probability of rejecting the null hypothesis when it is actually true. However, I am not sure what is meant by a two-tailed Type 1 error rate. Does the 'two-tailed' refer to the fact that the hypothesis test used is two-sided? Any insights are appreciated.
Examples of two-sided and one-sided t tests
Suppose you have data as follows: I used R to take a sample of size $n = 30$ from a normal distribution population with mean $\mu = 100.$ However, I happened to get a sample with mean $\bar X - 95.55,$ somewhat smaller than $\mu = 100.$
set.seed(1234) x = rnorm(30, 100, 15) summary(x) Min. 1st Qu. Median Mean 3rd Qu. Max. 64.81 86.85 92.49 95.55 103.62 136.24
I will do a t test of the null hypothesis $H_O: \mu =100$ against the two-sided alternative $H_a:\mu \ne 100.$ I want to test at the 5% level of significance.
t.test(x, mu = 100) One Sample t-test data: x t = -1.798, df = 29, p-value = 0.08259 alternative hypothesis: true mean is not equal to 100 95 percent confidence interval: 90.49593 100.61132 sample estimates: mean of x 95.55363
The P-value is $0.08259 > 0.05 = 5\%,$ so I do not reject at the 5% level. Even though I happened to get a sample that is a bit strange, its sample mean 95.55 is not enough different from $\mu_0 = 100$ to reject the null hypothesis.
The P-value is the probability under the density curve of Student's t distribution with 29 degrees of freedom of getting a T statistic farther from $0$ than the observed $-1.798.$ In the figure below that amounts to the sum of the areas in the two tails outside the vertical red lines.
However, if I decide (perhaps after seeing the small value of $\bar X)$ to do a left-sided test of $H_0:\mu=100$ against $H_a:\mu < 100,$ also at the 5% level. Then I get P-value $0.0413 < 0.05 = 5\%,$ so I reject the null hypothesis. In the figure, this is the area in the left tail (only) to the left of the solid red line. (The P-value of the one-tailed test is half of the P-value of the two-sided test.)
t.test(x, mu=100, alte="less") One Sample t-test data: x t = -1.798, df = 29, p-value = 0.0413 alternative hypothesis: true mean is less than 100 95 percent confidence interval: -Inf 99.75543 sample estimates: mean of x 95.55363
Even though both the two-sided and the one-sided test were at the same 5% level, we failed to reject $H_0$ for the two-sided test and rejected for the one-tailed test. This is because the criteria for rejection are different for the two tests. So, in a practical application, it is important to decide from the start (preferably before data are available) whether you need to do a two-sided or a one-sided test.
Notes: (1) In case you're familiar with R and want the R code for the figure, it is shown below.
hdr = "Density of T(29)" curve(dt(x, 20), -3.5, 3.5, lwd=2, ylab="PDF", xlab="t", main=hdr) abline(h=0, col="green2") abline(v=0, col="green2") abline(v=-1.798, col="red") abline(v=1.798, col="red", lty="dotted")
In case you want to see how P-values can be obtained in R, here they are:
1 - diff(pt(c(-1.798, 1.798), 29))  0.08259619 # 2-sided P-value pt(-1.798, 29)  0.04129809