Let's say I have a sample size established with alpha=0.05 and power=0.8 (based on time constraint for study).

The same sample size can be achieved with any of:

alpha = 0.001 and power = 0.31
alpha = 0.01 and power = 0.58
alpha = 0.25 and power = 0.95
alpha = 0.999 and power = 0.998

I want to know is what levels of risk to expect from the study of this size: what alpha I can aim for and what power I can hope to achieve.

Which combination of alpha and power do I adopt? Why?


One way to think about it is that if you plot power versus alpha, you obtain what is in essence an ROC curve (receiver operating characteristic). Typically, ROC curves are used to evaluate the capability of a diagnostic procedure such as cancer screenings. They weigh the probability of a true positive (i.e., power) against the probability of a false positive (I.e., alpha). An overall measure of the capability of the diagnostic test is the area under the ROC curve.

As for what alpha and power to choose, that depends on the relative consequences of attaining a false positive versus a false negative. The greater the relative cost of a false positive, the smaller the alpha you should choose.

| cite | improve this answer | |
  • $\begingroup$ But I don't understand the consequences. My problem is it seems the same sample has different alpha/power profiles SIMULTANEOUSLY. How is a study for which I arbitrarily chose alpha 0.05 any different from the hypothetical exact same study with same sample but alpha 0.01, since I effectively just converted that alpha to 0.05 by increasing power. Or to put another way, how can two researchers run a shared study with the same hypothesis and get one dataset, but one says "significant" while the other says "not even close", just based on their own arbitrary choice of an alpha/power combination. $\endgroup$ – Vlad Sep 21 '14 at 4:01
  • $\begingroup$ I've plotted the alpha and power values as you suggested. It's not surprisingly an exponential-looking curve with a "turn" at alpha 0.05 and power 0.8, followed by a sharp rise. After this point, an increase in power increases alpha drastically. This is a useful visualization. But you also said it's the "AREA under the curve" is the overall measure of the capability (the sum of all the alpha/power combinations in this case). Does this not contradict the idea that I must chose a particular POINT on this curve rather than consider the set of all possibilities. $\endgroup$ – Vlad Sep 21 '14 at 4:17
  • 2
    $\begingroup$ When you ANALYZE the data, you choose alpha (NOT power), based on the cost of a type I error vs. a type II error. Once the study is conducted, power is irrelevant. You use power in the PLANNING a stages to ensure that you have a good chance (i.e., power) of detecting an effect of predetermined size considered to be important, if it should exist. You do not choose an (alpha,power) pair at time of analysis. You choose SAMPLE SIZE - BEFORE the data are collected - so that a specified power is achieved with specified alpha, effect size, error SD, etc. $\endgroup$ – Russ Lenth Sep 22 '14 at 0:32
  • $\begingroup$ What I meant in the previous comment is that each of the two researchers picked a different power/alpha combination BEFORE the analysis and then each proceeded to analyze the same data and reach a different conclusion. $\endgroup$ – Vlad Sep 23 '14 at 1:04
  • $\begingroup$ OK, so? Two different political pundits on TV look at the same speeches and arrive at different conclusions. Two different radiologists look at the same mammogram and arrive at different diagnoses. In your example, neither of those researchers chose the power; they chose alpha - presumably because they had different assessments of the cost of making a type I error. Look again in your statistics texts - where do you see advice to choose the power of a test. You can't, because it is unknowable, because the true size of the effect is unknown. $\endgroup$ – Russ Lenth Sep 23 '14 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.