1
$\begingroup$

I was hoping to get some advice regarding an optimum statistical test. I've recently done some work looking at survival in a patient cohort after an intervention. I have the data on actual survival (AS) from this retrospective cohort. Using an outcome-predicting score (which is based on numerous factors eg age, type of disease) I have also calculated a predicted outcome (PS) for each patient.

Could someone advise on the best way to assess how tightly actual and predicted survival correlate? The ideal thing would be to show that cases with low actual survival also had low predicted survival, and vice versa.

Thank you in advance!

Improve this question
$\endgroup$
4
  • $\begingroup$ How about using some sort of generalized R squared measure? $\endgroup$
    – Demetri Pananos
    Commented Jul 30, 2021 at 17:04
  • $\begingroup$ Assuming survival is a binary variable in this context, a logistic regression model could tell you which of the predictors - if any - contribute significantly to your outcome prediction. You could also determine to which extent the model outperforms guessing by base rate (e.g. if on average 80% of patients survive, assume they all do.) Very low numbers (e.g. almost no survivors or almost all survivors) van render modelling tricky. Often, model outcome is evaluated by means of a contingency table (hits, misses, false alarms, correct rejections). $\endgroup$
    – KrisBae
    Commented Jul 30, 2021 at 17:13
  • $\begingroup$ What variables do you have? Are AS binary (so you have a classification problem) and the PS probabilities of survival? Or are both actual and predicting remaining lifetimes, i.e., numerical variables? (In both cases, you may need to deal with censoring.) $\endgroup$
    – Stephan Kolassa
    Commented Jul 30, 2021 at 17:19
  • $\begingroup$ Thank you for help so far. The prediction tool is already tested and validated (and published), so there isn't any need to test how significant each predictor is. Essentially the cumulative score with this tool gives a predicted overall survival in months. The only bits of data I'm looking to compare are actual survival (months) vs predicted survival (months). $\endgroup$
    – breadfruit21
    Commented Jul 30, 2021 at 21:52

1 Answer 1

Reset to default
1
$\begingroup$

The simplest (and perhaps easiest to understand) metric here is the concordance, a measure of discrimination. That's the fraction of comparable pairs of cases in which the predicted and observed order of events agree. Restricting to comparable pairs of cases gets around the problem posed by right censoring (cases where there was no event observed so you only have a lower limit to the event time). Comparable pairs of cases are all pairs of cases with event times, and all pairs involving 1 case with an event time and censored cases having censoring times greater than that case's event time. In R, that's implemented via the concordance() function in the survival package.

The next level of analysis is a calibration curve. As your model gives a predicted survival duration, plot observed survival against predicted survival in a way that deals with censoring. Say that you have 210 cases. Put them in order of predicted survival duration, and break them into 10 bins (21 cases each) of increasing predicted survival. For each bin, use the median of the predicted survival time within the bin as the horizontal-axis coordinate. Then, for that bin, get a Kaplan-Meier estimate of actual median survival and confidence interval, and plot along the vertical axis. (The Kaplan-Meier estimate handles the censoring problem for observed survival.) If the model fits well, the line connecting the 10 bin-median points should have a slope near 1.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Don't use arbitrary binning in getting a calibration curve. See the Cox model chapter of RMS course notes for a method for a smooth calibration curve with censored data. Then see the R rms package val.surv and calibrate functions. $\endgroup$
    – Frank Harrell
    Commented Jul 31, 2021 at 20:41
  • $\begingroup$ @FrankHarrell is there a way to use validate and calibrate in this situation, where the model is a published "tool [that] gives a predicted overall survival in months," built from data that are apparently unavailable to the OP, and the interest is in how well that published tool matches the OP's data? As I understand it, calibrate evaluates predicted survival probability at a given point in time. Here, the published "tool" evidently provides predicted survival times and the interest is in comparing those predictions against observed survival times in a new data set. $\endgroup$
    – EdM
    Commented Aug 1, 2021 at 16:30
  • $\begingroup$ Thank you for help so far - think I'm getting the hang of it. Plan at the moment is to calculate a correlation coefficient. Have seen some talk above about doing an R-squared goodness-of-fit test - what would this add that a correlation coefficient wouldn't? What I'd really like is to be able to say that where predicted survival is low, actual survival is also low, and vice versa. $\endgroup$
    – breadfruit21
    Commented Aug 1, 2021 at 17:28
  • 1
    $\begingroup$ @breadfruit21 a calibration plot like I suggest (or a continuous one, if that can be implemented as Frank Harrell suggests) would be the best way to show that "where predicted survival is low, actual survival is also low, and vice versa." A correlation coefficient on that plot would demonstrate the linear portion of the relationship, but there might be important non-linear portions. $R^2$ doesn't add anything if you use a simple linear fit. You can't just do correlation/$R^2$ on the raw survival data, as for the censored cases you only have a lower limit to the survival. $\endgroup$
    – EdM
    Commented Aug 1, 2021 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.