How to get the survival duration prediction for each individual in the data by using the Kaplan-Meier method?

I am trying to learn how to use the Kaplan-Meier survival estimator model in the lifelines package. The documentation says that the KaplanMeierFitter.fit function returns "a modified self, with new properties like 'survival_function_'." I checked what the survival_function_'s contents are - it seems to contain the average survival probability for all the players in the dataset at each time time interval. For example, in my dataset, there are 66 months and about 250,000 players (i.e., individuals whose death event we are trying to predict - 75% of them have had their deaths and the rest 25% is censored data, i.e., their deaths have not been observed), so the survival_function_ contains the following:

>>> kmf.survival_function_

KM_estimate
timeline
-1.0    1.000000
0.0     0.995473
1.0     0.779609
2.0     0.621312
3.0     0.508698
4.0     0.424205
5.0     0.366714
6.0     0.324090
7.0     0.289432
8.0     0.259339
9.0     0.234256
10.0    0.212542
11.0    0.192735
12.0    0.172880
13.0    0.157821
14.0    0.144604
15.0    0.132614
16.0    0.121743
17.0    0.112202
18.0    0.103710
19.0    0.095829
20.0    0.088811
21.0    0.082302
22.0    0.076773
23.0    0.071249
24.0    0.065752
25.0    0.060534
26.0    0.056082
27.0    0.051978
28.0    0.048073
...     ...
37.0    0.023696
38.0    0.020562
39.0    0.017846
40.0    0.015783
41.0    0.013817
42.0    0.012253
43.0    0.010645
44.0    0.009354
45.0    0.008186
46.0    0.007195
47.0    0.006274
48.0    0.005486
49.0    0.004656
50.0    0.003948
51.0    0.003391
52.0    0.002823
53.0    0.002352
54.0    0.002004
55.0    0.001655
56.0    0.001388
57.0    0.001114
58.0    0.000932
59.0    0.000707
60.0    0.000536
61.0    0.000343
62.0    0.000193
63.0    0.000080
64.0    0.000038
65.0    0.000016
66.0    0.000000

68 rows × 1 columns


It tells us the average survival probability of the entire population at each time period, taking both dead as well as censored players. It does not tell us the survival probability for each individual censored player, which is what I am interested in. How do I find that? It can be as detailed as, giving the survival probability for each individual player for each of the 66 months. Or, if that's not possible, I'm ok with having having the survival probabilities of each individual player at a fixed time in the future, say 3 months, or anything else (which is 1 of the 66 time periods).

In other words, instead of the output being a 66x1 vector of average survival probabilities, can I get an output matrix of dimensions txn, where t is the number of time periods and n is the number of censored players in the dataset, and the entry (i,j) is the survival probability of player i at time period j?

If this is not possible with the KM method, please feel free to suggest other methods where its possible to get the survival estimate for each individual. Thank you.

EDIT: I tried Cam Davidson Pilon's suggested way, and the prediction matrix gives the same exact predictions for each individual for a given time period:

This is not what I wanted; I expected a prediction matrix where the (i,j)'th entry (the survival probability for person j at time i) would be mostly unique. Whereas this just takes the survival_function_ values and copies them for each individual.

👋Hi author of lifelines here. So what you asking is possible. The Kaplan-Meier curve gives you is a distribution of possible durations, where duration is the time between birth and death. However, given a player has existed for $$N$$ months, you can condition the survival function on $$T > N$$ to get a better estimate.

Let $$S(t) = P(T \ge t)$$ be the survival function. We are curious about $$S(t | T \ge N)$$.

$$S(t | T\ge N) = \frac{P(T \ge t \text{ and } T \ge N)}{P(T \ge N)} = \frac{P(T \ge t)}{S(N)} = \frac{S(t)}{S(N)},\;\; t \ge N$$

So we simply need to divide the survival function by itself evaluated at the duration seen thus far.

In your use case, you could do something like:

predictions = pd.DataFrame(index=kmf.survival_function_.index)

for ix, row in alive_individuals.iterrows():
T = row['T']
predictions[ix] = kmf.survival_function_/kmf.survival_function_.loc[T]

# can't have probabilities great than 1.
predictions[predictions > 1.0] = 1.


This gives you the new survival function. However, in lifelines, there is another utility you can use. kmf.conditional_time_to_event_ computes these conditional survival functions and then takes the median time remaining. Output using some fake data I have:

          KM_estimate - Conditional time remaining to event
timeline
0.0                                                    56.0
6.0                                                    50.0
7.0                                                    49.0
9.0                                                    47.0
13.0                                                   43.0
15.0                                                   41.0
17.0                                                   39.0
19.0                                                   37.0
22.0                                                   34.0
26.0                                                   32.0
29.0                                                   29.0
32.0                                                   26.0
33.0                                                   27.0
36.0                                                   24.0
38.0                                                   22.0
41.0                                                   19.0
43.0                                                   17.0
45.0                                                   15.0
47.0                                                   13.0
48.0                                                   13.0
51.0                                                   10.0
53.0                                                    8.0
54.0                                                    7.0
56.0                                                    7.0
58.0                                                    5.0
60.0                                                    8.0
61.0                                                    7.0
62.0                                                    6.0
63.0                                                    6.0
66.0                                                    3.0
68.0                                                    1.0
69.0                                                    6.0
75.0                                                    0.0


So if a player lives until age 62, we expect 6 more months left (6 months being the median time to death, given the player lived to 62). That may help as well.

• Thanks for the prompt reply, Cam! About the line predictions[predictions > 1.0] = 1.0, wouldn't it be better to divide the predictions matrix by the max value in it? Something like, predictions /= max(map(max, predictions)). I say this because, I'm no statistician, but I feel the procedure by which we calculate probabilities should inherently have an upper bound of 1, isn't it? And if it somehow doesn't, maybe "normalizing" it by dividing it by the max value would make more sense than just truncating the decimal digits in the probabilities greater than 1, isn't it? Nov 23, 2018 at 6:49
• This way, at least we preserve the relative magnitudes among the numbers. What I mean is, if there was a number in predictions that was big, say around 10. Then by simply making it 1 and not changing the other numbers, we lose this info. Whereas, if we divide all the numbers for that player by 10, we know that something funny is happening at that time duration for this player. What do you say? Nov 23, 2018 at 6:56
• Also, how do I check the prediction accuracy? How do I validate the prediction? Nov 23, 2018 at 7:13
• dividing by the max doesn't make sense in this context. Note I chose 1.0 because that's what we observed! For a given individual who lived to atleast month N, their probability of dieing prior to month N was 0.0, so the survival curve should be 1.0 at those points. Note that the predictions dataframe has the same index as the survival function too. Nov 23, 2018 at 14:27
• Hi Cam. I tried your suggested method, and I've edited the question details in the end to include the results. I did not get what I expected, which was (mostly) unique probabilities for each individual for each time period. Instead what I got is the exact same survival probabilities for each individual at each time period. Please advice. In other words, this just takes the survival_function_ values and copies them for each individual. Nov 26, 2018 at 8:13

One condition of use K-M method to estimate the survival probability is that all subjects follow the ONE survival curve. Your situation is: When you use K-M method, you agree with one survival curve assumption, but after that you want to get the individual level survival curve. In fact, it is impossible.

If you think it is necessary to fit the survival curves for different groups of subjects, for example, male vs female, you need to run K-M multiple times.

Especially, in survival analysis, we assume that censoring has no relation with survival probability after censoring. For example, at time 25, 2000 were censored, and 20000 were still followed. If you believe (based on your knowledge/experience) that 2000 censored subjects would survival longer (or shorter) than the 20000 followuped subjects, then even K-M methods is invalid.

• Well, about your last statement "If you believe (based on your knowledge/experience) that 2000 censored subjects would survival longer (or shorter) than the 20000 followuped subjects, then even K-M methods is invalid.", I thought we use survival analysis for these tasks exactly for the reason that it can handle the case that the survived population can live much longer, potentially, than the dead population. Isn't that correct? Otherwise, why not simply use linear regression to predict the survival duration? Nov 23, 2018 at 7:18
• censored at time 25 means the subjects are still alive at time 25, but after time 25, their disappeared, and you do not know when they die. For all of survival analysis, the common assumption is: Given the subjects are alive at time t, and some of them censored (disappeared), and some of them still are under the follow-up, the survival probabilities after time t are the same between two groups. Nov 23, 2018 at 15:14