I was asked this question about the demonstration of power.

There are two groups of data, say control and treatment. If conducting a test( say a linear regression including other factors, but the response variable was log-transformed before entering the regression), the results shows no significant difference between the control and the treatment group.

A colleague manipulated the data by dividing all treatment outcome variable by 2, then doing the same analysis and now there is significant difference between the two groups. Then he argued that if the treatment group data was lower by a factor of 2, this method could have identified the difference. And this shows the power of the analysis or of the test.

I do think there is nothing wrong with this as a way to demonstrate a what would be case. But I feel it is odd to manipulate data to show the power. It is conceptually similar to minimum detectable difference, but the manipulation ignored other factors by treating all the y's from the treatment group in the same way. Of course if you shift all the data in the same direction far enough with respect to the sample size, then the difference would have been identified as significant.

Is this a real added value to the analysis and worth of some space in a journal publication?

  • 2
    $\begingroup$ Although the question is interesting, I do not see that it has much to do with power, since the conclusion is conditional on one particular dataset whereas power depends on a spectrum of assumptions about the hypothesized underlying process--and has nothing at all to do with any particular dataset. I also do not see that this procedure says much of interest about the data themselves, but that may be due to my ignorance about the details of the experiment and the data. You might want to search our site for related posts. $\endgroup$
    – whuber
    Feb 20 '15 at 23:16
  • $\begingroup$ @whuber actually for me retrospective power is an empty concept and the minimum detectable difference(MDD) tells nothing more than the confidence intervals. However, I have to admit that in the area I work at, they use MDDs very often to make a conclusion like if the difference would be this big, the test would have picked this effect as a significant effect (assuming the observed variance is the "true" variance). This holds for simple tests. However, when there are more components in the model, this MDD can only be generalized to specific values for different combinations of the predictors. $\endgroup$
    – Zhenglei
    Feb 20 '15 at 23:36
  • $\begingroup$ And I agree it is not power, at best it is an indicator of the power. $\endgroup$
    – Zhenglei
    Feb 20 '15 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.