What is the relation of the significance level alpha to the type 1 error alpha?

Question

In statistical hypothesis testing we decide on and set the acceptable probability of error or significance level α (alpha) to a value that fits our theory. Traditionally alpha is .1, .05, or .01.

When we calculate the power function g of the parameter we test for, we recieve the distribution of the probability of two errors: the Type 1 error α (alpha) and the Type 2 error β (beta). In a graphical representation of this function, alpha is the value below the graph, beta is the value above the line: α = g(p) and β = 1 - g(p), with p being the parameter we are interested in.

Here is an example:

The red line is α_max for H₀: p ≤ 0.4 and H₁: p > 0.4; the blue line is β for a sample p̂ = 0.5

How do the probability of error (alpha) and the Type 1 error (alpha) relate to each other?

Or am I just getting confused over two unrelated values having the same name (alpha)?

Answer 1

When doing a continuous test and all the assumptions hold then the 2 alphas are exactly the same thing. For example if I perform a t-test on a mean and set my significance level to alpha=0.05 (or anything else) and the null hypothesis is true (the only time I can make a type I error) and all the other assumptions hold, then the probability of me making a type I error is exactly alpha.

The case where there can be a difference is when dealing with discrete probabilities. For example, I want to test if a coin is fair and plan to flip the coin 10 times. I set alpha = 0.05 as is traditional, that means that I will only reject the null hypothesis (prob=0.5) if out of 10 flips I see 0, 1, 9, or 10 heads (if I reject at 2 or 8 heads then the probability of a type I error is greater than 0.05, so would not be allowed). But if the coin is fair, then the probability of rejecting (type I error) is not 0.05, but is around 0.022 (from memory, but not that hard to compute if you want to do it yourself).

There are other cases where the true probability of a type I error may be less than the level set (one sided hypothesis is one example), so sometimes you will see alpha or the significance level defined as an upper bound on the probability of a type I error.

Answer 2

When you are doing a statistical test, the significance level is set at the desired type I error level (alpha). So the concepts you are asking about are basically the same thing - both are fixed by design to the same value. The p-value is calculated from the data and is different from the alpha value, and may be why you are getting confused.

When doing a power calculation, typically the type I error value is fixed, as is either the available sample size, or the desired type II error level (beta). Given an expected effect size (or in the case of your graph, it appears to specify an expected proportion) the non-specified value is calculated (either necessary sample size, or available type II error level - used to get power = 1-beta).

I'm not familiar with the graph you've provided, but it appears to show how the expected effect size changes the available beta level, and demonstrate the relationship between alpha and beta. There is a natural trade-off between type I and type II error, in that if you improve one, you will worsen the other. Type I error is being calculated in this graph, but in general is not something that is calculated from your data.