Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have seen many cases where type I errors are accounted for (denoted by an alpha value) in various research articles. I have found it rare that a researcher will take into consideration the power, or the type II error.

Type II errors can be a big deal right? We have accidentally rejected the alternative hypothesis when it was actually false. Why are alpha values emphasized so much instead of beta values?

When I took first year statistics, I never was taught beta—only alpha. I feel that these two errors should be treated equally. Yet, only alpha seems to be emphasized.

share|improve this question
1  
+1 The reason is that traditionally, the Type I error (aka, $\alpha$ or the significance level) is fixed first, and then the test is constructed such as to minimize the Type II error (equivalently, such as to maximize the power). A helpful article on wikipedia to understand the issue is the one on Uniformly Most Powerful (UMP) tests, en.wikipedia.org/wiki/Uniformly_most_powerful_test – Jeremias K Feb 28 at 18:03
    
Interesting! Thanks! – sponge_knight Feb 28 at 18:04
3  
You are wrong about "we have accepted the null hypothesis" -- we never accept it. We either "reject null hyp", or "fail to reject null hyp", but never accept null hyp! – caveman Feb 28 at 18:46
    
blasted - that skimmed past me. Thanks for pointing that out. – sponge_knight Feb 28 at 18:52
1  
Be careful not to confuse your own experience with the entire field of statistical literature; you can hardly infer the content of material you haven't read. – Glen_b Feb 28 at 23:05
up vote 7 down vote accepted

This is a good question. Let me begin with a couple of clarifications:


I think you are (unfortunately) right that less attention is paid to power and type II errors. While I think the situation is improving in biomedical research (e.g., funding agencies and IRBs often reqire power analyses now), I think there are a couple of reasons for this:

  1. I think power is harder for people to understand than simple significance. (This is in part because it depends on a lot of unknowns—notably the effect size, but there are others as well).
  2. Most sciences (i.e., other than physics and chemistry) are not well mathematized. As a result, it is very hard for researchers to know what the effect size 'should' be given their theory (other than just $\ne0$).
  3. Scientists have traditionally assumed that type I errors are worse than type II errors.
share|improve this answer
1  
As always, enlightening - especially for the non-mathematized :-) ... I love this wording... I wonder if you could expand a bit on the third point... Is there any basis for this bias. I know it's true, but why do you think this is the case... Is it because it's about the trophy of the p-value, and nothing else matters? – Antoni Parellada Mar 6 at 0:47
    
Thanks, @AntoniParellada. I'll think about what more I could add. – gung Mar 6 at 1:09
2  
I would clarify point 3) why scientists think type I errors are worse. The null hypothesis is usually some sort of "status quo", e.g. the effect of this brand new drug is 0. We like the status quo, and the burden of proof is on the researcher to prove otherwise. Thus, we want to limit Type I error, i.e. we wrongly reject the status quo. IMO, this attachment to the status quo is just philosophical. If you want to change my opinion, you'll have to prove it. – Heisenberg Mar 6 at 1:12
1  
In practice, one could easily think of cases where type II error matters a lot more, i.e. the cost of not rejecting the null is high. For example, if mankind faces a zombie epidemic, I'm sure the attitude would be "try any drug even if it may not work" rather than "you have to prove that it works before we use it". – Heisenberg Mar 6 at 1:17
    
Adding to @Heisenberg: In cases where type II errors matter most, one should consider switching between point hypothesis tests and equivalence test. In your example, one would have to prove that a proposed worcester sauce at least doesn't worsen the zombie epidemic. Then the error rates change their role and the most important error rate is fixed by design again. Also, if you have some cost estimate of wrong decisions, one should consider a decision rule that minimizes risk and does not (necessarily) fix a particular type I error rate. – Horst Grünbusch Mar 17 at 13:22

The reason is that we simply don't know the actual type II error rate and we never will. It depends on a parameter we usually don't know. In turn, if we would know this parameter, we would not need to do a statistical test.

However, we can plan an experiment such that a specific type II error rate is met given some alternative is true. This way, we would choose a sample size that does not waste resources: Either because the test doesn't reject in the end or because already a much smaller sample size would have been sufficient to reject the hypothesis.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.