# Interpreting the result of post hoc power calculation with powerCT.default for proportinal hazards model

I have 2 samples of data both with over 800000 instances. . In both samples, ~ 75% of people have the exposure.

In the first sample 0.3 % of people have an outcome and in the second 5% have an outcome. The variability is due to the seasonality of the outcome (the flu).

I ran 2 Cox models for 2 samples, (also adjusted by other covariates). In the first model, the adjusted HR for the exposure was 1.01 (0.87-1.12), and in the second 1.40 (1.35-1.47).

I did the post hoc power analysis for the first model:

powerCT.default(nE = 191324, nC = 611287, pE = 0.00322, pC = 0.00225,RR = 1.01, alpha = 0.05)

And the result is 0.4241588, which is lower than 0.8, the commonly used threshold.

I am not sure how to interpret this result - the HR1.01 is insignificant, so what does it mean that the power is 0.4 for the insignificant result?

I tested if the power would be over 0.8 if I had more outcomes

powerCT.default(nE = 191324, nC = 611287, pE = 0.0322, pC = 0.0225,  RR = 1.01,alpha = 0.05)


Here, the power would be over 0.99

Does this mean that the insignificant ratio for my first model is due to the insufficient number of outcomes, not due to the lack of true difference between the group with the exposure and no exposure, which I cannot assess?

While in this second, made-up example there is enough power so the insignificant HR is due to the lack of the difference between the exposed and not exposed groups.

Is that correct? Also, is there any way to estimate how many outcomes I should have to get a significant result for the differences between groups?

• You should not do post hoc power analysis. See, e.g. this thread and several others. Commented May 8 at 12:25
• I have read this thread. But what is the other way to check if the insignificant effect is due to the lack of difference between exposed and not exposed or due to the insufficient number of cases/outcomes? Also, does it mean that all power calculations do not make sense? How other can it be done?
– Milo
Commented May 8 at 13:01
• That's what a priori power calculations are for. Post-hoc power calculation is essentially just presenting the obtained p-value in another way. Commented May 8 at 13:28
• Post hoc power tells you that power was sufficient to detect "significant" effects, and insufficient to detect "insignificant" ones. Post hoc power does not give you any new information. That is why you calculate power prior to the experiment, feeding in an effect size you would be sorry to miss. Commented May 8 at 13:39
• Insignificant effects are always due to a combination of effect size, choice of p value, and sample size. If you sample 100,000,000 cases, then even a really, really small difference will be significant. Doing a priori power analysis lets you find out how many observations you will need to have a reasonable chance of finding significance for a given effect size. Commented May 8 at 13:47

I don't know how to calculate power/sample size for proportional hazards model because I don't use them. But generally, to conduct a priori power calculations for a simple design, you need to provide the power calculation function the effect size you're interested in, your desired power level, and your alpha level. For instance, let's say I want to know how many participants I need to detect a small between-group difference. I can run power calculations for a simple between-subject ANOVA in R pwr:

library(pwr)
pwr.anova.test(k=2, f=.10, sig.level=0.05, power=.80)

#I have k=2 groups, a small effect size in Cohen's f metric = .10, and I want 80% power to detect the effect with alpha = .05).

Balanced one-way analysis of variance power calculation

k = 2
n = 393.4057
f = 0.1
sig.level = 0.05
power = 0.8

NOTE: n is number in each group

#I'd need 394 participants per group


This kind of power calculation can be easily done with many free R packages, G*Power software (also free to use), and there are even web calculators.

However, when you have a more complex design, such as interactions between continuous variables, clustered or longitudinal data or, I assume, survival analysis data, simulation is usually the way to go. When simulating power, you typically need to provide all parameters that you ultimately plan to enter in your model, and then you test several different sample sizes and see which one gives you the desired level of power for detecting the parameter of interest.

Here is one introductory source. I recommend checking out different online sources and experimenting on your own starting from simple designs.

Nowadays, there are also some web apps that do the simulation for you for some complex designs, such as InteractionPoweR and SuperPower. However, I don't think such an app exists for proportional hazards data. The way to go with that seems to be powerSurvEpi package I mentioned in a comment.

• There are some functions in Frank Harrell's Hmisc package to help design survival studies: cpower() for two-sample designs (allowing for dropins, cross-over, and non-compliance), ciapower() for interaction designs, and spower() for some simulations. The simsurv package allows for simulations of more complicated scenarios. (+1)
– EdM
Commented May 8 at 14:29