Actual vs. predicted survival comparison

Question

I have a set of patients and their actual 1- and 5-years survival. I have also used their data with a certain commonly utilised medical score, that calcualtes survival probability for 1- and 5-years (for example 75% and 55% respectively). I'd like to compare both survival rates.

I did calculate the mean survival probability for all patients at 1- and 5-years as the mean of predicted survival probabilities. I then calculated the mean actual survival by using 100% if alive at 1 year and 0% if dead at 5 years. I then compared the means of both groups with a t-test.

I have a feeling that what i am doing is grossly incorrect and goes against all rules of statistics, however i have not find any solution of my problem anywhere. Maybe someone can help me? R packages and codes are welcome.

Answer 1

You're evidently trying to check the calibration of the medical score against real obsrvations, which is a good idea. There are a couple of problems with the specifics of your approach, however.

First, by using the average predicted and average actual survival over all your patients, you're ignoring all the patient-specific information. (I'm assuming that the medical-score prediction values differ among patients.) Second, even if you were to compare the overall averages, the t-test isn't usually the best test for comparing proportions, as the assumption of a normal distribution of values is unlikely to hold. (That said, if your survival probabilities are on the order of 50-75%, the error in using a t-test might not be so big; the problems are largest near the extremes of probability.)

There are a few ways you could try to improve your efforts. One classic measure of the quality of a probabilistic model that takes into account individual cases is the Brier score, which is similar to the mean-square error in linear regression. Here, you could calculate for each patient the squared difference between predicted survival probability and actual survival (1 for alive, 0 for dead), then take the average squared difference over all patients, at each of 1 year and 5 years. You will have to be careful, however, if some of your patients are still alive but haven't yet reached either of the 1-year of 5-year levels. It's easy to make a mistake and code a patient as "alive" at 5 years even if she's only been followed out to 3 years. A small Brier score indicates a good model.

As a further step, you could plot the medical-score predictions against the observed survival to get a calibration curve. With 190 patients, you might put them in order of predicted survival probabilities and then break them down into 10 groups of increasing predicted-survival order. Then do the averages that you proposed in your question within each of those 10 groups and plot the predicted against the observed survival probabilities for the 10 groups. Do that for each of 1-year and 5-year survival. If the model is well calibrated, a line through the points on the plot should be close to the line of identity. Such a plot can illustrate particular probability-prediction regions in which the model has problems.

These tasks are implemented, for example, in the val.prob() function in the R rms package, along with other ways to evaluate probabilistic model predictions against observed events. See this answer for more details.

Answer 2

See

Pocock, Stuart J., et al. "Predicting survival in heart failure: a risk score based on 39 372 patients from 30 studies." European heart journal 34.19 (2013): 1404-1413.

also

Uno, Hajime, et al. "On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data." Statistics in medicine 30.10 (2011): 1105-1117.

for some examples.

In particular, Figures 3, 4 and 5 from the first paper show very good ways of describing the relationship between the risk score and the observed survival rates. These are easy enough to make and to understand.

Another thing you could do that is more complicated is the C statistic described in the second paper. The idea is to take every pair of subjects and count the proportion of times that the risk score is in the same direction as the observed survival. In other words, if patient A has a worse risk score than patient B, count it as 1 if patient A is known to have died before patient B (as the risk score predicts). Count as 0 if B is known to die before A (opposite of what the risk score predicted). Count 1/2 if it is not known which died first. Add those all together and divide by the number of pairs. If that proportion is close to 1, then the risk score almost perfectly predicts survival. You can get a confidence interval etc. This has the advantage that it puts everything into a single number. But, the graphs like in the first cited paper show a lot of good information.