# What does it mean to determine the power of your experiment via simulation?

What does it mean to determine the power of your experiment via simulation? Does it mean to run the experiment over and over again and count how many times a specific set of null hypotheses are rejected given that they are false?

• You might want to take a look at this: simulation-of-logistic-regression-power-analysis-designed-experiments, & this: calculating-statistical-power. Apr 10 '13 at 14:21
• Not to run an actual experiment repeatedly, but a 'simulation experiment' - to use simulation of random variables to emulate the characteristics of the assumed testing situation under whatever effect size(s) you're interested in - to have computer-generated ersatz experiments that tell you about the proportion of rejections under a given set of assumptions. But yes, you count the proportion of rejections at a particular effect size to compute the power at that effect size. Apr 10 '13 at 14:32
• Just out of curiosity, for a split plot experiment, there are two experimental units, whole plot units, and split plot units. Are there multiple power terms for the subplot and whole plot effects? Apr 10 '13 at 14:47
• Be aware that a "false null hypothesis" is not a definite state of affairs and so requires further specification. For instance, when comparing two proportions $p$ and $q$, the null hypothesis $p=q$ leads to a definite distribution of a test statistic, whereas its negation $p\ne q$ leaves us unable to do anything quantitative until we specify exactly how much $p$ differs from $q$. Thus power depends on "effect size". Among other things, this means that computing power via simulation typically requires many separate simulations across a range of effect sizes.
– whuber
Apr 10 '13 at 16:01