# Calculating Statistical Power after Multiple comparison tests?

I'm currently using the multtest package in R, and I am curious to see how the statistical power holds up in a FWER vs. FDR based corrections.

Say for example:

pval <- c(0.018,0.034,0.726)
mt.rawp2adjp(pval,"Bonferroni")
mt.rawp2adjp(pval,"TSBH")


Is there anyway I can get the details of the power of the test retrospectively, using either this package or any other package? Or is what I am asking not making anysense?

## 1 Answer

What you are asking doesn't really make sense. Statistical power is a function of your effect size, sample size, and significance criterion. It's a measure of how frequently you'll accept the null hypothesis when you should actually reject it. Power can't be computed from just a list of p-values.

At any rate, it's not really useful to compute the power of your test after you've applied it and gotten p-values. If you find significant p-values, your test was "powerful enough", otherwise, there's either no effect or your test didn't have enough power.

• The reason I'm after this exercise is that I have read FDR based adjustments preserve the power of the test much more than FWER  now before you say that I must use the right tool for the right job, and FWER and FDR are actually asking two different questions etc.. I want to show myself that the FDR correction does indeed preserve more power...or is this something I can not realistically do?  Shaffer J.P. (1995) Multiple hypothesis testing, Annual Review of Psychology 46:561-584, – Rover Eye Jul 17 '17 at 14:47
• If you have benchmark data where you know what variables actually DO have an effect, you can directly estimate power by calculating how many false negatives you get for each test. – Nuclear Wang Jul 17 '17 at 14:50
• If needed, I can use randomly generated normally distributed data, so that shouldn't be a problem. How do I go about doing this? – Rover Eye Jul 17 '17 at 14:52
• Power is a useful thing to compute in advance of a study, using hypothetical effect sizes that represent targets of practical interest. Power is not useful to calculate retrospectively; it's an empty exercise, sort of like wondering if you included enough respondents in your pre-election poll after the election is over and the votes are counted. – rvl Jul 18 '17 at 2:35