I'm currently using the multtest package in R, and I am curious to see how the statistical power holds up in a FWER vs. FDR based corrections.

Say for example:

pval <- c(0.018,0.034,0.726)

Is there anyway I can get the details of the power of the test retrospectively, using either this package or any other package? Or is what I am asking not making anysense?


What you are asking doesn't really make sense. Statistical power is a function of your effect size, sample size, and significance criterion. It's a measure of how frequently you'll accept the null hypothesis when you should actually reject it. Power can't be computed from just a list of p-values.

At any rate, it's not really useful to compute the power of your test after you've applied it and gotten p-values. If you find significant p-values, your test was "powerful enough", otherwise, there's either no effect or your test didn't have enough power.

  • $\begingroup$ The reason I'm after this exercise is that I have read FDR based adjustments preserve the power of the test much more than FWER [1] now before you say that I must use the right tool for the right job, and FWER and FDR are actually asking two different questions etc.. I want to show myself that the FDR correction does indeed preserve more power...or is this something I can not realistically do? [1] Shaffer J.P. (1995) Multiple hypothesis testing, Annual Review of Psychology 46:561-584, $\endgroup$ – Rover Eye Jul 17 '17 at 14:47
  • $\begingroup$ If you have benchmark data where you know what variables actually DO have an effect, you can directly estimate power by calculating how many false negatives you get for each test. $\endgroup$ – Nuclear Wang Jul 17 '17 at 14:50
  • $\begingroup$ If needed, I can use randomly generated normally distributed data, so that shouldn't be a problem. How do I go about doing this? $\endgroup$ – Rover Eye Jul 17 '17 at 14:52
  • $\begingroup$ Power is a useful thing to compute in advance of a study, using hypothetical effect sizes that represent targets of practical interest. Power is not useful to calculate retrospectively; it's an empty exercise, sort of like wondering if you included enough respondents in your pre-election poll after the election is over and the votes are counted. $\endgroup$ – rvl Jul 18 '17 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.