0
$\begingroup$

How to compare the time-dependent precision recall (PR) receiver operating curve (ROC) values for two cox regression models at multiple time points?

To compare two time-dependent AUC values, I would use the compare function of the timeROC R library. For a long time I thought DeLong's test as an alternative but it was originally developed for comparing standard (binary classification) AUROCs, where the outcome is binary and independent of time. Time-dependent AUROCs, however, incorporate censoring and time-to-event information.

The PRROC R library works well to calculate time-dependent ROC values for PR curves (no statistical comparison) and the pr.test function of the usefun R library to compare two PR values (not time-dependent!). However, there are no well-documented libraries to combine both so to statistically compare the time-dependent PRROCs.

Any ideas how to implement this in R or python? Could not find any other threads addressing this specific issue.

$\endgroup$
4
  • $\begingroup$ Welcome to Cross Validated! Are your Cox models in a clinical or similar context (e.g., human disease/death) or for some other type of application? It's hard for me to think of a clinical situation where true negatives don't matter so that PR curves might be useful. Also, do you have any references showing that "to statistically compare the time-dependent PRROCs" is a good way to compare models? Frank Harrell, who developed the c-index (effectively AUROC with time-invariant predictors) for Cox models, doesn't recommend it for model comparison. $\endgroup$
    – EdM
    Commented Nov 4 at 21:12
  • $\begingroup$ Thanks EdM! You are correct, this is for a clinical context. I usually report both AUROC and PRROC values as these complement nicely one another, but only the first one can also be adapted to time-dependent setting. In this case, I don't think Frank Harrell would necessary as we are comparing how two categorical models predict time-to-event (here the critique is on thresholding continuous to categorical variables fharrell.com/post/class-damage). In addition, I think using AUROC and PRROC are more easily interpreted here than for example C-index (which measures rank). $\endgroup$
    – obruzzi
    Commented Nov 5 at 7:28
  • $\begingroup$ AUROC also only measures rank. It’s equivalent to the probability that the model puts 2 randomly chosen cases into the correct order; see Wikipedia. Harrell’s c-index is simply the same probability when analysis is restricted to comparable cases when censoring is involved. If there’s no censoring, AUROC and the c-index are identical. Any rank-discrimination index like AUROC has limited value for comparing models. Likelihood-based methods are typically more sensitive. $\endgroup$
    – EdM
    Commented Nov 5 at 11:26
  • $\begingroup$ Thanks, appreciate the comment. I agree that to assess with one metrics model performance, likelihood-based methods make sense. Still, I find these a bit unintuitive to interpret clinical benefit, which might just reflect my lack of knowledge and how the field usually employs metrics to compare which model better predicts mortality and why (e.g. better to identify early deaths? high FP or TN classifications etc). I will think this over but leave the question open in case there are other suggestions! $\endgroup$
    – obruzzi
    Commented Nov 6 at 10:57

1 Answer 1

1
$\begingroup$

General considerations

The most sensitive ways to compare Cox (or other) regression models overall are based on (partial) likelihood methods. It's also important to evaluate the calibration of the model, in terms of how well predicted and observed outcomes agree (at some particular point in time for a survival model). See this post by Frank Harrell.

Receiver operating characteristic (ROC) or precision-recall (PR) curves aren't so sensitive as those preferred methods for model comparisons. They are both rank-based displays that don't take calibration into account.

To a great extent the limitations of ROC curves carry over to PR curves, as if you have an ROC curve you can generate the corresponding PR curve from the corresponding numbers of true positives (TP), true negatives (TN) and false positives (FP) at each point (predictor value) along the ROC curve. "Recall" for a PR curve is just a synonym for the "true positive rate" or "sensitivity" plotted on the vertical axis of the ROC curve. Precision is TP/(TP+FP). (True negatives, TN, aren't used in PR curves, so you can't go from PR to ROC.)

The area under the ROC curve (AUROC) in binary-outcome models, often used nevertheless to compare models, is equivalent to the probability that the model puts 2 randomly chosen cases into the correct order. Harrell's c-index for survival outcomes is effectively equivalent, as it's the fraction of comparable cases (taking censored event times into account) that are put in the correct order. In the post linked above, even Harrell doesn't recommend using the c-index for comparing models, just for documenting a particular model's ability to discriminate cases.

The area under a PR curve (AUPR) is considered an "average precision," but that interpretation needs to be tempered a bit by the curve's nature. There's a whole region of PR space that is unachievable for a given data set, determined by the event probability. See for example Boyd et al., Proceedings of the 29th International Conference on Machine Learning, Edinburgh, Scotland, UK, 2012. That sets a lower limit to the "average precision" that needs to be taken into account.

When there's a low event probability, differences between models are admittedly more evident in AUPR than in AUROC; see Saito and Rehmsmeier, PLoSone 10:e0118432, 2015 and Zhao et al., Diagnostic and Prognostic Research 5:13, 2021. I'm not aware, however, of any evidence that the AUPR is superior to proper model calibration in this low-probability regime.

If you really want to compare PR curves, despite the above, I'd be reluctant to depend on any theoretical, model-based statistic. Bootstrapping would seem to be the best choice. Even that might be problematic, as the lower limit to AUPR might prevent AUPR from being a pivotal quantity suitable for bootstrap estimation. I suspect that "BCa" (bias and acceleration corrected) bootstrapping would be needed.

Time-dependent curves

As noted above, once you have an ROC curve you can generate the corresponding PR curve. That's also true, in principle, for time-dependent ROC curves. Specifics with censored event times will depend on the type of smoothing or weighting used to generate the curve and on which type of time-dependent curve is generated. Kamarudin et al. review those types of time-dependent curves and smoothing/weighting methods in BMC Medical Research Methodology (2017) 17:53. See Figure 1 for a simple outline of how the numbers of cases evaluated depend on the predictor value along the ROC curve and the type of curve; you'll need that to parcel out the TP and FP numbers if those aren't immediately provided by the software you use.

Note that the timeROC package that you cite has a SeSpPPVNPV() function that can be used to estimate time-dependent "sensitivity" (aka "recall") and "positive predictive value" (PPV, aka "precision") that you would need to generate PR curves directly from your data. Again, if you really want to compare AUPR values between models, bootstrapping (with the above cautions) would be my choice.

$\endgroup$
2
  • $\begingroup$ Thanks EdM for the detailed comment and excellent points and references. Are there any evidence that the lower sensitivity tests (AUROC, AUPR) actually would mislead over using likelihood-based metrics suggested by Frank Harrell. For example, in the post fharrell.com/post/addvalue Frank demonstrated in the simplistic use case that blood cholesterol add "0.16 to 0.19" predictive value over the base model (age + gender) in predicting occurrence of coronary artery disease. Most clinicians (me included) would prefer to know how many patients benefit from measuring cholesterol. $\endgroup$
    – obruzzi
    Commented Nov 7 at 19:33
  • $\begingroup$ Therefore, I provocatively suggest that what is statistically the most correct way to measure something might not be the most easier result to interpret and apply to real-life use cases :) I really don't like either AUROC and AUPC as they do not take calibration into account as you point out, but they might be easier to use with nonparametric modelling frameworks (for instance decision-tree-based models) where e.g. R^2 might not be easy to extract, let alone interpret, to compare models. For these cases, bootstrapping of AUPR and AUROC might have their place, and thanks for suggesting these! $\endgroup$
    – obruzzi
    Commented Nov 7 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.