I have, probably, a simple problem.

I've finished analysing the results of an observational prospective study conducted in our unit. In this study I evaluated if a specific biomarker is independently associated with mortality. Additionally, I have tried to test if a new cut-off for this biomarker could perform better than an already, let's say, "literature validated" cut-off in a Cox adjusted survival analysis.

My problem is that I had only 39 incident outcomes and the number of the a priori chosen variables for adjustment in the Cox analysis is 7. To avoid the problem of overfitting due to the low number of incident outcomes, I performed bootstrapping validation, in order to determine the confidence intervals for estimating ß in the Cox analysis.

However, one of the persons that is reviewing my analysis says that I should also perform a post-hoc power analysis for this study, considering also the little difference between the two cut-offs (the previously cut-off is 15, and my cut-off is 17.4). I have read about the fact, and probably you will confirm it, that performing a post-hoc power analysis is not considered correct, but I have to do it. Is it correct to use this formula?enter image description here

But then, how can I find if my study was powered enough to detect a difference between two cut-offs?


First, as Russ Lenth has put it:

You've got the data, did the analysis, and did not achieve "significance." So you compute power retrospectively to see if the test was powerful enough or not. This is an empty question. Of course it wasn't powerful enough -- that's why the result isn't significant. Power calculations are useful for design, not analysis.

Either you found a significant difference or you didn't. (You don't tell us in your question.)

So the reviewer requiring this post-hoc analysis evidently is not an expert at statistical design and analysis of biomarker studies. Whether this person is at your own institution or is a reviewer evaluating your manuscript for publication, get some local help from a statistician to address this concern.

Second, looking for cutoffs in biomarker values (or in any continuous variable) is typically not a good idea. Even if clinical decisions end up being yes/no, a biomarker value is only one part of the clinical decision-making process. If this is a quantitative biomarker, its value presumably has some continuous (not necessarily linear) relation to outcome and it's much more important to identify that relation, to incorporate with other clinical variables for decision-making (for example in a nomogram), than to set an arbitrary cutoff.

Third, with only 39 events you are hard pressed to evaluate even 3 or 4 variables related to outcome. Usual rules of thumb are 10 to 20 events needed per variable. With only 39 events in your prospective observational study, it sounds like the entire study was under-powered for 7 predictor variables, not just your attempt to distinguish a cutoff of 17.4 from one of 15.

Fourth, although the attempt to minimize over-fitting by bootstrapping is a good start, it's not clear that you did the bootstrapping in a way that would accomplish your goals. The implication of your question is that you found 17.4 to be a better cutoff than 15 based on analysis of the data you collected, rather than on some theoretical basis, and then did bootstrap comparisons based on the 15 versus 17.4 cutoffs. That's probably not the right way to proceed even if getting a better cutoff is a worthwhile goal. Bootstrapping should encompass the entire process, which in your case includes the process for choosing the "better" cutoff.

Finally, the particular formula you cite seems to be for incidence studies rather than survival studies, in which time-to-event is important. For purposes of study design rather than for post-hoc analysis, consider an on-line tool for simple power evaluations, or learn the tools provided by a computing environment like R.

  • $\begingroup$ First of all, in my analysis I have found a significant difference. I totally agree with your second comment. Even as a continuous variable, this biomarker is associated with all cause mortality. But, it is important to find what patients are at an increased risk, not only to say to a patient that an increase level of this biomarker increases your risk of dying. My entire cohort included 212 patients with 66 outcomes - and this is the cohort from which I have derived the 17.4 cut-off (using the Martingale residuals). $\endgroup$ – Dimitrie Jun 22 '15 at 20:53
  • $\begingroup$ Unfortunately, only in 157 patients I had also performed an echocardiography assessment. In this subgroup I have only 33 events and I had performed the mentioned analysis (with bootstrapping). Although it seems rather simple (the on-line tool you mentioned), I don't know how to use it. I'm not sure what accrual time means (is it the inclusion period?), what to put in the follow-up interval (the median, mean or the longest follow-up interval that a patient had) or in the median time to failure (as you can see I don't have 50% deaths). Thank you again. $\endgroup$ – Dimitrie Jun 22 '15 at 21:10
  • $\begingroup$ Power analysis of a study that you have already done is meaningless, and in the long-run both you and the project will be better off if you can convince the person reviewing your analysis that such is the case. $\endgroup$ – EdM Jun 23 '15 at 13:15
  • $\begingroup$ For future reference, the on-line tool I cited is for pre-study power analysis in a situation with a defined start date, having participants added continually until an "accrual time" after the start, and followed up until the "follow-up interval" after the start (as I recall; maybe after the "accrual time"--that's easy to check by trying some values). It assumes an exponential distribution of times between events, so extrapolation of available data to estimate median survival time would be OK to start. Or do a Google search for "survival analysis power online" for more tools. $\endgroup$ – EdM Jun 23 '15 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.