1. To perform calibration of a Cox regression model (i.e. assessing for the agreement between the predicted and the observed outcome), what is the best method to present the accuracy of the model in predicting the actual event?

  2. As far as I understand, we can calculate the actual outcome probability by observing the number of events that occurred in a number of subjects with similar/same predicted probability from the Cox model. To perform the above calculation, do we stratify the predicted risk into several groups (<15%, 15-30%, 30-45% etc.), and within each risk group we use the number of subjects as the denominator for the calculation of actual outcome?

  3. What method do we use to compare the predicted outcome with the actual outcome? Is it good enough if we simply present the predicted and actual risk% in each risk group in table format? Can rms package in R do all calibrations for you?

  4. Can we use pec::predictSurvProb() to give the absolute risk of event for each individual? Can we specify the time point for the risk/hazard function for each individual to be at the ENDPOINT of follow up?

  5. When interpreting the results, do we use the mean follow up period (in years) as the time point on which the predicted risk and actual risk are based? (E.g. Individual A has 30% risk of event at 6.5 years (mean follow up period))

  6. Is the goodness-of-fit test for Cox regression (Gronnesby and Borgan test) simply a means for calibration for cox regression? Or does it mean something else?

  7. To compare models with net reclassification, how many subjects and outcomes do we need for such method to become valid?

  1. Cox models do not predict outcomes! "Best" methods depend on whether you obtain a risk score (as with Framingham) or absolute risk (as with Gail Breast Cancer Risk). You need to tell us exactly what you're fitting

  2. With absolute risk prediction, you can split groups according to their risk deciles and calculate proportions of observed vs. expected outcome frequencies. This is basically the Hosmer Lemeshow test. But, in order to use this test, you need to have an absolute risk prediction! You cannot, say, split the groups by risk score deciles and use the empirical risk as the risk prediction, this strips off too much information and leads to some counter intuitive results.

  3. The bioconductor package has a suite of tools related to ROC analyses, predictiveness curves, etc.

  4. Nowhere in Ulla's package is mention made of estimating smoothed baseline hazard estimates. This is necessary to obtain risk prediction from survival models... because of censoring! Here's an example of that method being applied. I would accept no less from the package.

  5. No, don't use mean follow up. You should report total person years follow-up, along with censoring rate, and event rate. The Kaplan Meier curve kinda shows you all of that.

  6. I'm sure Sir David Cox is not fond of G&B's test. The power of the Cox model is that it can give consistent inference without necessarily having predictive accuracy: a tough concept for many to grasp. Tsiatis' book "semiparametric inference" has a lot to say about this. However, if you aim to take the Cox model one step further and create predictions from it, then I think the G&B test is very good for that purpose.

  7. Reclassification indices are proportions of individuals being shuffled into different (more discriminating) risk categories comparing two competing risk prediction models (see Pencina). It's important to realize (Kerr 2011) that you can calculate confidence intervals for this value... not using the bootstrap (or any limit theory treating the model as fixed) but using the double bootstrap (bootstrap sample, refit model, bootstrap sample again, calibrate models).

| cite | improve this answer | |
  • 1
    $\begingroup$ (+1) Nice answer. $\endgroup$ – chl Nov 8 '13 at 18:02
  • $\begingroup$ @ AdamO - (b) Any R packages would allow the performing of Hosmer Lemeshow test or G&B's test for Cox regression? $\endgroup$ – user32454 Nov 9 '13 at 10:27
  • $\begingroup$ @ AdamO - (a) I am trying to assess the incremental value of a biomarker in risk modelling by fitting models with that biomarker and models without that biomarker and compare the predicted ABSOLUTE RISK(%) of developing adverse cardiovascular outcomes in models with and without the biomarker (in a cohort with a mean follow up period of 6.5 years). $\endgroup$ – user32454 Nov 9 '13 at 10:30
  • $\begingroup$ Do read the article I linked from Gail. Getting absolute risk from Cox models requires a smoothed estimate of the BL hazard, as done here. With that, HL test code popped up in R forums long ago. Also see the work done by Janes and Pepe which I think is very insightful. I haven't seen the G&B test elsewhere. Any chance you can write it up on your own? $\endgroup$ – AdamO Nov 9 '13 at 15:49

@ AdamO is more knowledgeable than me, but if what you are trying to do is see if adding a biomarker to a model improves the model then this is a case of "nested" models. What's needed is:

  1. Define the null hypothesis as H0: No improvement in risk prediction
  2. Apply the likelihood ratio test to calculate twice the change in the log-likelihood statistic (D).
  3. This difference D follows a chi-squared distribution with n degree of freedom (n=1 in this case if you are adding just 1 biomarker). You look up D in a chi-squared table to find a p value.

I've played with assessing the added value of a biomarker risk to a logistic regression model. If you are interested in there are some new statistical metrics (including net reclassification - often poorly applied) and a way of visualising them that are far more informative than merely comparing AUCs (c stats). See my paper Pickering, J. W., & Endre, Z. H. (2012). New Metrics for Assessing Diagnostic Potential of Candidate Biomarkers. Clinical Journal Of The American Society Of Nephrology, 7, 1355–1364. doi:10.2215/CJN.09590911

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.