The estimated power of a given test can be reported with a table or with a curve in a plot. I need to compare the power of two permutation tests by simulation and I was wondering how to create a good-looking plot. Suppose that I've estimated the power of the tests using different effect sizes. As a consequence, for each test I have the set of points: $(\delta_1,\hat{power_1}),(\delta_2,\hat{power_2}),....(\delta_k,\hat{power_k})$
How can I interpolate exactly these points to get a smooth curve? Can loess be used? Do I need to use a smoothing spline? What is the usual method?
I'd like to get a result similar to this one:
Thank you.
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ This lacks some detail. When you say "estimated power" where is the "estimation" (rather than calculation) part coming in? Are you simulating in order to obtain power at a given n (and delta)? (presumably at a specific effect size) -- which would lead to trying to estimate a power curve from sampled rejection rates. Or are your power values actually calculated exactly and you just need to interpolate? $\endgroup$– Glen_bJan 29, 2016 at 0:36
-
$\begingroup$ Actually, I don't know if I'm estimating or calculating the power of the test. I'm comparing the power of two permutation tests by simulating the dataset. I try different values for sample size and effect size and then I calculate/estimate the power of the tests with those values. $\endgroup$– John MJan 29, 2016 at 7:41
-
$\begingroup$ If you're calculating rejection rates by simulating samples, you'd get (slightly) different answers if you did it again (took a different set of samples), so that's estimation rather than calculation. It makes at least some difference to the discussion so the information should be in your question $\endgroup$– Glen_bJan 29, 2016 at 8:25
-
$\begingroup$ If you're varying both effect size and $n$ you don't have a curve but a surface (or at the least, a set of curves). Did you want power curves or a power surface? If you want power curves, do you want curves against effect size across several different sample sizes? $\endgroup$– Glen_bJan 29, 2016 at 12:18
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Note that if you're finding power across both effect sizes and across sample sizes you'd have a power surface rather than a power curve.
We've also got estimates of rejection rates from simulation, so rather than interpolation there will be some smoothing of noisy estimates.
The rejection counts at any combination of effect size and sample size will be binomial so we could fit GLMs or GAMs to do the smoothing (though often an adequate fit can be obtained via weighted least squares for example).
For most statistics in common use, as sample size gets large $\sqrt{n}\cdot \delta$ (where $\delta$ is effect size for some power) tends to be nearly constant, which simplifies the task of smoothing in the sample-size direction.
For one-tailed tests, in a number of common cases the normal quantiles of the power is nearly linear in effect size; this suggests fitting a probit link, possibly with natural cubic splines or a locally linear smooth. (In some other situations logistic quantiles might be better approximated by a line, so local linearity in a logit model would be a good choice in such situations.)
For two tailed tests you'll sometimes tend to have near linearity away from the null but it may be nearly quadratic close to effect size 0.
If you have the time to run a lot of simulations at each of a large number of different $n$ and $\delta$ it's less important how you smooth -- sometimes you can just use sample proportions and linear interpolation -- in at least some situations this may be enough.
