What does it mean to determine the power of your experiment via simulation? Does it mean to run the experiment over and over again and count how many times a specific set of null hypotheses are rejected given that they are false?

  • 2
    $\begingroup$ You might want to take a look at this: simulation-of-logistic-regression-power-analysis-designed-experiments, & this: calculating-statistical-power. $\endgroup$ Apr 10, 2013 at 14:21
  • 1
    $\begingroup$ Not to run an actual experiment repeatedly, but a 'simulation experiment' - to use simulation of random variables to emulate the characteristics of the assumed testing situation under whatever effect size(s) you're interested in - to have computer-generated ersatz experiments that tell you about the proportion of rejections under a given set of assumptions. But yes, you count the proportion of rejections at a particular effect size to compute the power at that effect size. $\endgroup$
    – Glen_b
    Apr 10, 2013 at 14:32
  • $\begingroup$ Just out of curiosity, for a split plot experiment, there are two experimental units, whole plot units, and split plot units. Are there multiple power terms for the subplot and whole plot effects? $\endgroup$
    – phil12
    Apr 10, 2013 at 14:47
  • 1
    $\begingroup$ Be aware that a "false null hypothesis" is not a definite state of affairs and so requires further specification. For instance, when comparing two proportions $p$ and $q$, the null hypothesis $p=q$ leads to a definite distribution of a test statistic, whereas its negation $p\ne q$ leaves us unable to do anything quantitative until we specify exactly how much $p$ differs from $q$. Thus power depends on "effect size". Among other things, this means that computing power via simulation typically requires many separate simulations across a range of effect sizes. $\endgroup$
    – whuber
    Apr 10, 2013 at 16:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.