In statistical hypothesis testing we decide on and set the acceptable probability of error or significance level α (alpha) to a value that fits our theory. Traditionally alpha is .1, .05, or .01.

When we calculate the power function g of the parameter we test for, we recieve the distribution of the probability of two errors: the Type 1 error α (alpha) and the Type 2 error β (beta). In a graphical representation of this function, alpha is the value below the graph, beta is the value above the line: α = g(p) and β = 1 - g(p), with p being the parameter we are interested in.

Here is an example:

sample power function

The red line is αmax for H0: p ≤ 0.4 and H1: p > 0.4; the blue line is β for a sample p̂ = 0.5

How do the probability of error (alpha) and the Type 1 error (alpha) relate to each other?

Or am I just getting confused over two unrelated values having the same name (alpha)?

  • 1
    Traditionally, $\alpha = 0.05$ rather than $\alpha = 0.005$. – ocram Jun 13 '13 at 9:57
  • @ocram Thank you, yes, I had one 0 too many in all my "traditional alpha values". I edited my question accordingly. – user14650 Jun 13 '13 at 10:00
  • You seem to be talking about the same thing both times; in some circumstances, you may see people distinguish between level and significance, but in simple cases that most people encounter in practice, they refer to exactly the same thing. – Glen_b Jun 13 '13 at 10:52
  • How can they be the same thing? The significance level / probability of error is defined by the statistician to be a certain value, e.g. 0.05, while the probability of the Type 1 error is calculated from the values he observes and different from 0.05. – user14650 Jun 13 '13 at 12:33
up vote 4 down vote accepted

When doing a continuous test and all the assumptions hold then the 2 alphas are exactly the same thing. For example if I perform a t-test on a mean and set my significance level to alpha=0.05 (or anything else) and the null hypothesis is true (the only time I can make a type I error) and all the other assumptions hold, then the probability of me making a type I error is exactly alpha.

The case where there can be a difference is when dealing with discrete probabilities. For example, I want to test if a coin is fair and plan to flip the coin 10 times. I set alpha = 0.05 as is traditional, that means that I will only reject the null hypothesis (prob=0.5) if out of 10 flips I see 0, 1, 9, or 10 heads (if I reject at 2 or 8 heads then the probability of a type I error is greater than 0.05, so would not be allowed). But if the coin is fair, then the probability of rejecting (type I error) is not 0.05, but is around 0.022 (from memory, but not that hard to compute if you want to do it yourself).

There are other cases where the true probability of a type I error may be less than the level set (one sided hypothesis is one example), so sometimes you will see alpha or the significance level defined as an upper bound on the probability of a type I error.

  • I understand. If I did not flip the coin n = 10 times, but n → ∞ times, the calculated true alpha would approach set alpha. Is that correct? – user14650 Jun 14 '13 at 5:55
  • @what, yes that is correct. – Greg Snow Jun 14 '13 at 17:09

When you are doing a statistical test, the significance level is set at the desired type I error level (alpha). So the concepts you are asking about are basically the same thing - both are fixed by design to the same value. The p-value is calculated from the data and is different from the alpha value, and may be why you are getting confused.

When doing a power calculation, typically the type I error value is fixed, as is either the available sample size, or the desired type II error level (beta). Given an expected effect size (or in the case of your graph, it appears to specify an expected proportion) the non-specified value is calculated (either necessary sample size, or available type II error level - used to get power = 1-beta).

I'm not familiar with the graph you've provided, but it appears to show how the expected effect size changes the available beta level, and demonstrate the relationship between alpha and beta. There is a natural trade-off between type I and type II error, in that if you improve one, you will worsen the other. Type I error is being calculated in this graph, but in general is not something that is calculated from your data.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.