What does it mean to determine the power of your experiment via simulation? Does it mean to run the experiment over and over again and count how many times a specific set of null hypotheses are rejected given that they are false?
2
$\begingroup$
$\endgroup$
-
2$\begingroup$ You might want to take a look at this: simulation-of-logistic-regression-power-analysis-designed-experiments, & this: calculating-statistical-power. $\endgroup$ – gung♦ Apr 10 '13 at 14:21
-
1$\begingroup$ Not to run an actual experiment repeatedly, but a 'simulation experiment' - to use simulation of random variables to emulate the characteristics of the assumed testing situation under whatever effect size(s) you're interested in - to have computer-generated ersatz experiments that tell you about the proportion of rejections under a given set of assumptions. But yes, you count the proportion of rejections at a particular effect size to compute the power at that effect size. $\endgroup$ – Glen_b♦ Apr 10 '13 at 14:32
-
$\begingroup$ Just out of curiosity, for a split plot experiment, there are two experimental units, whole plot units, and split plot units. Are there multiple power terms for the subplot and whole plot effects? $\endgroup$ – phil12 Apr 10 '13 at 14:47
-
1$\begingroup$ Be aware that a "false null hypothesis" is not a definite state of affairs and so requires further specification. For instance, when comparing two proportions $p$ and $q$, the null hypothesis $p=q$ leads to a definite distribution of a test statistic, whereas its negation $p\ne q$ leaves us unable to do anything quantitative until we specify exactly how much $p$ differs from $q$. Thus power depends on "effect size". Among other things, this means that computing power via simulation typically requires many separate simulations across a range of effect sizes. $\endgroup$ – whuber♦ Apr 10 '13 at 16:01
