9
$\begingroup$

This question is raised by my supervisor and I don't know how to explain.

Usually the accepted confident level is 0.95 which means that the probability of type 1 error is 5%. But the usually accepted power is 0.8 (Cohen, 1988) which means that the probability of type 2 error is 20%. Why we can accept higher probability of type 2 error than type 1 error? Is there any statistical reason behind that?

He also asked the physical meaning of power = 0.8 (why it is selected as a criteria) which I also have no idea to explain it.

And when we use power analysis to design the experiment we may select effective size 0.3, 0.5 or 0.8 to represent the small, medium and large effects. And my supervisor asked why these numbers are selected. My understanding is that these numbers are suggested based on experience. He immediately asked me what is the experience. I am really frustrating about such questions. My major is not statistics and I need to spend a lot of time on such questions, which I think may not be meaningful. Can any one suggest if such questions are really meaningful or not? If yes, how shall I find out the answer.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Neither the 5% type one error rate nor the 80% figure for power are universal. Indeed, I doubt your average physicist has even heard of Cohen. $\endgroup$
    – Glen_b
    Sep 18, 2014 at 10:05
  • 1
    $\begingroup$ A jaded, cynical, but likely partially correct answer is "that's what reviewers in your field will demand." $\endgroup$
    – whuber ♦
    Sep 18, 2014 at 17:01

1 Answer 1

Reset to default
8
$\begingroup$

Neither the 5% type one error rate nor the 80% figure for power are universal. For example, particle physicists tend to use a "5 sigma" criterion that corresponds to a notional type I error rate that is roughly on the order of one in a million. Indeed, I doubt your average physicist has even heard of Cohen.

But one reason why the two error rates you quote should be different would be that the cost of the two error types would not be the same.

As to why the type I error rate is often taken at 5%, part of the reason (some of the historical background for the convention) is discussed here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Many thanks! (1) First, I can understand that the 5% type one error and 80% of the power is not universal.(2)Regarding "the cost of the two error types would not be the same", could you elaborate more? (3) The information of the 5% of type I error is very useful. By any chance, do you have similar information for type II error (power = 0.8)? Thanks again and sorry that I cannot vote for your answer because I do not have sufficient reputation. $\endgroup$
    – user2230101
    Sep 18, 2014 at 11:01
  • 1
    $\begingroup$ (2) Consider the difference between failing to identify an effect that's present and identifying as present an effect that's absent. In some circumstances the consequences of the second are minor (leading to an inexpensive but useless treatment), but in other situations it may be much more significant; conversely the type II error may miss a crucial improvement, or a less important one - such as missing a cure for Ebola, vs missing a hair dye that lasts a few days longer). (3) The 80% power is much less widespread. I don't have an equivalent background for it; you already know about Cohen. $\endgroup$
    – Glen_b
    Sep 18, 2014 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.