Is it possible to infer the power from the distribution of a statistic?

For instance, let's assume we are dealing with a Wald statistic, calculated from 2 survival $\beta$ coefficients. Under the hypothesis of given means and variances of the coefficients, it is possible to calculate the distribution of the statistic:

Wald statistic density

In this example, there is 2% of the Area Under the Curve of the Wald statistic distribution (black curve) that is greater than 0.

Would the proportion of the AUC that is greater than -0.05 be the power of the test at a 2-sided alpha=0.05?
(Or most likely some other value that my ingenuous, haphazard guess of -0.05)

Since we are assuming some mean and variance already, I'm not sure if we are still under either H0 or H1.

  • $\begingroup$ Could you explain what you mean by "AUC"? That has a standard meaning in statistics that does not appear to apply here. $\endgroup$ – whuber Jun 21 at 15:48
  • $\begingroup$ @whuber AUC means Area Under the Curve. I'm indeed not talking about AUROC if that is what you were thinking about. $\endgroup$ – Dan Chaltiel Jun 21 at 15:50
  • 2
    $\begingroup$ AUC is so associated with ROCAUC in machine learning that I do not think we can use those letters for a regular integral. $\endgroup$ – Dave Jun 21 at 15:52
  • $\begingroup$ I edited the post to explain the abbreviation $\endgroup$ – Dan Chaltiel Jun 21 at 16:00

The general answer to your question is: to calculate power, you need to know two distributions: the distribution of the test statistic when the null hypothesis is true, and the distribution of the statistic under the specific alternative hypothesis. Add to that the usual testing assumptions, i.e. the sidedness, the alpha-level (which determines the critical value). I don't mention the sample size here, because that should be folded into the calculation of the test statistic distributions, so to speak.

Having the mean and variance of those distributions only matters if the distributions are two parameter families, such as normal distributions. But test statistics under alternative hypotheses usually add a "non-centrality" parameter to the mix, which often behaves like a skewness coefficient, so rarely will those parameters make sense by themselves (without any supporting theoretical results).

  • $\begingroup$ Thanks! My Wald statistic is a special case where 2 beta coefficients are considered, the null hypothesis being that those 2 coefficients are equal. We can expect that the variance should be approximately the same under both the null and the alternative hypothesis. It is thus possible to draw both distributions of different mean and equal variance. How would you calculate the power then? (Of course, under the hypothesis that everything is normally distributed) $\endgroup$ – Dan Chaltiel Jun 21 at 17:27
  • 1
    $\begingroup$ You say "beta coefficients". Are you talking about regression? Are the two "beta coefficients" from two independent models? Is this linear regression? It sounds, possibly, like you're reinventing the wheel and you basically have a simple case of calculating the power from a t-test. $\endgroup$ – AdamO Jun 21 at 17:46
  • $\begingroup$ Actually, this is a rather complicated design in which we calculated beta coefficients from the expected survival of multiple groups in order to compare 2 hazard ratios. I enhanced the image to visualize the whole scene. $\endgroup$ – Dan Chaltiel Jun 21 at 18:44
  • $\begingroup$ Be it complicated or no, you gotta boil it down so casuals can understand, rather than patch layer by layer. The test for a difference in survival is log rank test. Are you aware of the technique? Have you read about the log rank test and power? $\endgroup$ – AdamO Jun 21 at 19:09
  • $\begingroup$ Yes, I am aware of the log-rank test but it requires events, while I only have distributions of the Wald statistic. That is why my question is not focused on the design, I think the question would be the same if I had the distribution of 2 means, as the Wald test is quite similar to the t-test for that matter. $\endgroup$ – Dan Chaltiel Jun 21 at 19:26

Finally, I came to an answer to this question on this website: https://stats.idre.ucla.edu/other/gpower/power-analysis-for-paired-sample-t-test/.

Here is a screenshot of their interface:


H0 is the red curve (difference == 0) and H1 is the blue curve (difference != 0).

The red curve is used to get the critical t value (here, a t-test) and the power is then computed using the area under the blue curve between t and -t.

They also add a parameter to alter the normality in the alternative hypothesis, but I couldn't manage to understand what.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.