22
$\begingroup$

As I understand it, I need to know at least three aspects (out of four) of my proposed study in order to conduct power analysis, namely:

  • type of test - I intend to use Pearson's r and ANCOVA/Regression - GLM
  • significance level (alpha) - I intend to use 0.05
  • expected effect size - I intend to use a medium effect size (0.5)
  • sample size

Could anyone recommend a good online power calculator that I can use to do a priori power calculation. (Can SPSS do a priori power calculation?)

I have come across GPower but I am looking for a simplier tool!

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Unfortunately SPSS package does not include a module for power analysis. IBM SPSS company sells a separate program for power analysis. $\endgroup$ – ttnphns Jan 17 '12 at 14:51
  • 6
    $\begingroup$ I'd give GPower a chance. With 20 or 30 minutes exploring it, you'll probably find it's very manageable--at least for procedures like correlation, not necessarily for a complicated regression model. $\endgroup$ – rolando2 Jan 17 '12 at 22:52
  • $\begingroup$ Thanks! Is there a user-friendly guide available on GPower? $\endgroup$ – Adhesh Josh Jan 18 '12 at 3:45
  • $\begingroup$ This looks like it is for a grant application. These are vexing to produce and to evaluate. For well used experimental designs (genome-wide association studies for example) there may be well documented specialised calculators. Otherwise, I think G. Jay Kerns answer is the right way to go with the following addition: while you are at it you should simulate a range of the most important parameters and present a graph. $\endgroup$ – Leo Schalkwyk Jan 24 '12 at 9:55
42
$\begingroup$

This isn't an answer you are going to want to hear, I am afraid, but I am going to say it anyway: try to resist the temptation of online calculators (and save your money before purchasing proprietary calculators).

Here are some of the reasons why: 1) online calculators all use different notation and are often poorly documented. It is a waste of your time. 2) SPSS does offer a power calculator but I've never even tried it because it was too expensive for my department to afford! 3) Phrases like "medium effect size" are at best misleading and at worst just plain wrong for all but the simplest research designs. There are too many parameters and too much interplay to be able to distill effect size down to a single number in [0,1]. Even if you could put it into a single number, there's no guarantee that Cohen's 0.5 corresponds to "medium" in the context of the problem.

Believe me - it is better in the long run to bite the bullet and teach yourself how to use simulation to your benefit (and the benefit of the person(s) you're consulting). Sit down with them and complete the following steps:

1) Decide on a model that is appropriate in the context of the problem (sounds like you've already worked on this part).

2) Consult with them to decide what the null parameters should be, the behavior of the control group, whatever this means in context of the problem.

3) Consult with them to determine what the parameters should be in order for the difference to be practically meaningful. If there are sample size limitations then this should be identified here, as well.

4) Simulate data according to the two models in 2) and 3), and run your test. You can do this with software galore - pick your favorite and go for it. See if you rejected or not.

5) Repeat 4) thousands of times, say, $n$. Keep track of how many times you rejected, and the sample proportion $\hat{p}$ of rejections is an estimate of power. This estimate has standard error approximately $\sqrt{\hat{p}(1 - \hat{p})/n}$.

If you do your power analysis this way, you are going to find several things: A) there were a lot more parameters running around than you ever anticipated. It will make you wonder how in the world it's possible to collapse all of them into a single number like "medium" - and you will see that it isn't possible, at least not in any straightforward way. B) your power is going to be a lot smaller than a lot of the other calculators advertise. C) you can increase power by increasing sample size, but watch out! You may find as I have that in order to detect a difference that's "practically meaningful" you need a sample size that's prohibitively large.

If you have trouble with any of the above steps you could collect your thoughts, well-formulate a question for CrossValidated, and the people here will help you.

EDIT: In the case you find that you absolutely must use an online calculator, the best one I've found is Russ Lenth's Power and Sample Size page. It's been around for a long time, it has relatively complete documentation, it doesn't depend on canned effect sizes, and has links to other papers which are relevant and important.

ANOTHER EDIT: Coincidentally, when this question came up I was right in the middle of writing a blog post to flesh out some of these ideas (otherwise, I might not have answered so quickly). Anyway, I finished it last weekend and you can find it here. It is not written with SPSS in mind, but I'd bet if a person were clever they might be able to translate portions of it to SPSS syntax.

| cite | improve this answer | |
$\endgroup$
  • 8
    $\begingroup$ +1 Good answer. It's worth pointing out the drawbacks of simulation. (The alternative is that power curves can be computed mathematically.) Simulation becomes unwieldy when many parameters (such as effect size and sample size) have to be manipulated or when you are seeking some threshold value, such as a minimum sample size. Even an approximate exact expression for the power can be valuable for indicating in general how the power behaves and for identifying initial solutions that can be polished with a little bit of simulation. $\endgroup$ – whuber♦ Jan 17 '12 at 16:19
  • 2
    $\begingroup$ @whuber Thanks, and you are absolutely right. Your comment reminds me that there's often addt'l uncertainty in the null/alt parameters (scant info, crummy pilot studies, etc.) which adds another layer of complexity to the simulation approach. This is another benefit of the mathematical approach. $\endgroup$ – user1108 Jan 17 '12 at 17:01
  • 2
    $\begingroup$ Instead of fixing the values of the unknown parameters it is useful to simulate them by assigning a prior distribution on these parameters and then to get a "prior power" (this is not a Bayesian approach, in spite of the concept of prior distribution, because we simulate the result of the frequentist test) $\endgroup$ – Stéphane Laurent Jan 17 '12 at 19:52
  • 6
    $\begingroup$ There are two problems with simulation: Learning it (this one is soluble) and getting step 3 done. In my experience, none of my clients would be willing to do 3). Many have trouble specifying ANY effect size whatsoever. To ask them to specify the parameters in (say) a multiple regression equation would be .... well, they wouldn't know how to answer, even if they know the meaning, they won't be willing to specify. $\endgroup$ – Peter Flom Jan 17 '12 at 20:36
  • 2
    $\begingroup$ Stephane yes, you are right, and that was what I meant by the extra layer I was trying to communicate. @Peter Sigh! yes, I've encountered this, too. I try to talk about means, standard errors, etc and then work out as much of the math as I can afterward. Part of it is a communication barrier which is sometimes a challenge. The unwillingness part is even tougher, though. It used to be that I would give up and try to fill in the blanks myself, but it rarely worked out well. That is, the answer's essentially a shot in the dark with a blindfold on and standing backward. $\endgroup$ – user1108 Jan 17 '12 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.