11
$\begingroup$

I am trying to understand why large amounts of censoring (ie. many patients are censored) is undesirable in Survival Analysis.

As a proof of concept, suppose there are 5 patients - all patients enter the study at the same time:

  • patient1 has event at t1
  • patient2 has event at t2
  • patient3 drops out of the study at t3
  • patient4 has event at t4
  • and when the study is over at t5, patient 5 has not had the event
  • t5 > t4 > t3 > t2 > t1

(Semi Parametric Approach) Here is my attempt to write the model and likelihood for a Cox-PH regression in this situation:

$$ h(t|X) = h_0(t) \exp(\beta^T X) $$ $$ L(\beta) = \prod_{i: \delta_i = 1} \frac{h(t_i|X_i)}{\sum_{j: t_j \geq t_i} \exp(\beta^T X_j)} $$ $$ L(\beta) = \frac{h(t_1|X_1)}{\sum_{j: t_j \geq t_1} \exp(\beta^T X_j)} \times \frac{h(t_2|X_2)}{\sum_{j: t_j \geq t_2} \exp(\beta^T X_j)} \times \frac{h(t_4|X_4)}{\sum_{j: t_j \geq t_4} \exp(\beta^T X_j)} $$

$$ L(\beta) = \frac{h(t_1|X_1)}{\exp(\beta^T X_1) + \exp(\beta^T X_2) + \exp(\beta^T X_3) + \exp(\beta^T X_4) + \exp(\beta^T X_5)} \times \frac{h(t_2|X_2)}{\exp(\beta^T X_2) + \exp(\beta^T X_3) + \exp(\beta^T X_4) + \exp(\beta^T X_5)} \times \frac{h(t_4|X_4)}{\exp(\beta^T X_3) + \exp(\beta^T X_5)} $$

(Parametric Approach) Here is my attempt to write the model and likelihood for a AFT model in this situation (note that the likelihood is based on the distribution of $\epsilon$ and not the survival times $T$ . I have heard that if we pick $T$ to have distributions such as Exponential or Weibull, then $\epsilon$ results in Extreme Value Distribution such as the Gumbel Distribution ):

$$ \log(T) = \mu + \beta^T X + \sigma \epsilon $$

$$ L(\mu, \sigma, \beta) = \prod_{i=1}^{n} \left[ f\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{\delta_i} \left[ 1 - F\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{1-\delta_i} $$

$$ L(\mu, \sigma, \beta) = \left[ f\left( \frac{\log(t_1) - \mu - \beta^T X_1}{\sigma} \right) \right] \times \left[ f\left( \frac{\log(t_2) - \mu - \beta^T X_2}{\sigma} \right) \right] \times \left[ 1 - F\left( \frac{\log(t_3) - \mu - \beta^T X_3}{\sigma} \right) \right] \times \left[ f\left( \frac{\log(t_4) - \mu - \beta^T X_4}{\sigma} \right) \right] \times \left[ 1 - F\left( \frac{\log(t_5) - \mu - \beta^T X_5}{\sigma} \right) \right] $$

So in the Cox-Ph and AFT model, how is inference (e.g. perhaps it results in high variance, high bias, non-consistency, larger sample sizes to achieve compared results vs smaller sample sizes with lesser censoring) and parameter estimation negatively affected when large numbers of the patients are censored? Does the mathematical optimization become difficult (e.g. incomplete matrix rank, matrix inverses not defined, non-identifiable model)?

$\endgroup$
1

7 Answers 7

15
$\begingroup$

I'm going to give a slightly different answer from the previous responses, that also has a more visual tint. For simplicity, suppose we want to estimate the survival function up to 1 year. To show how censoring impacts the analysis, I am going to use nonparametric bounds.

The bounds are the best/worst cases on the survival function given the observed data that are going to place no parametric assumptions. The bounds represent the extremes of what censoring is capable of doing to our estimates given the observed data. For the upper bound on the survival function, we will assume that all censored individuals do not ever have the event (up to 1 year). For the lower bound, we will assume that all censored individuals immediately had the event after they were censored.

The following is some code to generate survival times for $n=10000$ observations with different extents of censoring. In the zero case, there is no censoring. In the first and second cases, more censoring occurs.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from lifelines import KaplanMeierFitter

n = 10000
t_max = 1
t = 0.7 * np.random.weibull(1.5, size=n)   # Event times
t = np.where(t >= t_max, t_max, t)         # Admin censoring at 1 year
c0 = 2                                     # Censoring (none)
c1 = 1.5 * np.random.weibull(2., size=n)   # Censoring (some)
c2 = 0.9 * np.random.weibull(0.9, size=n)  # Censoring (more)

For c0, there is no censoring. The following code is for the bounds (since no censoring, the bounds are the same as the point estimate of the Kaplan-Meier)

t0_star = t                           # New variable for t
delta0 = np.where(t >= t_max, 0, 1)   # Event indicator (all events expect admin censor)

# Using Kaplan-Meier, since equivalent to CDF here
km0 = KaplanMeierFitter()
km0.fit(t0_star, delta0)
km0_St = km0.survival_function_  # Survival function

The following is code to get the bounds for censoring in case 1 (c1).

# Setting up data we get to observed in this case
t1_star = np.min([t, c1], axis=0)
delta1 = np.where((t <= c1) & (t < t_max), 1, 0)

# Upper bound computation
t1_staru = np.where(delta1 == 0, t_max, t1_star)  # All never events
km1u = KaplanMeierFitter()
km1u.fit(t1_staru, delta1)
km1u_St = km1u.survival_function_

# Lower bound computation
delta1l = np.where(t1_star < t_max, 1, delta1)  # All events at censoring times
km1l = KaplanMeierFitter()
km1l.fit(t1_star, delta1l)
km1l_St = km1l.survival_function_

# Merging bounds into single data object for plotting
bounds = pd.merge(km1l_St, km1u_St, left_index=True, right_index=True, how='outer')
bounds.ffill(inplace=True)

# Plotting the bounds
plt.fill_between(bounds.index, bounds.KM_estimate_x, bounds.KM_estimate_y, step='post')
plt.show()

When we apply this for each of the scenarios, we can get the following plot censoring bounds

So, as you can see, the bounds get wider the more censoring that occurs. Large amounts of censoring is bad in survival analysis because we have less information in the data. To compute point estimates for the survival function (with Cox, AFT, Kaplan-Meier, etc. models), we need to rely on assumptions regarding how censoring occurs. The more censoring that occurs, the more we have to 'lean' on those assumptions. So, that is why less censoring is generally better.

$\endgroup$
5
  • 1
    $\begingroup$ thank you! this is a really cool simulation! I had a similar idea to this one but did not know how to put it into words ... I think you have done a wonderful job helping me understand this: confidence intervals in the presence of more censoring become wider (i.e. wider confidence intervals are undesirable). but why does this happen? i wonder if its possible to look at the mathematical formulas for the variances of survival estimates (either S(t), H(t) or the beta coeffecients) and see how more censoring results in larger widths? $\endgroup$ Jan 4 at 16:51
  • 3
    $\begingroup$ These are not confidence intervals. These bounds are a separate idea. The bounds are the 'best' and 'worst' possible point estimates consistent with the data, which is separate from confidence intervals. Confidence intervals would narrow as N goes up, but the bounds would not be narrowed. Confidence intervals are related to the amount of censoring, but that is beyond what I presented here $\endgroup$
    – pzivich
    Jan 4 at 17:09
  • 1
    $\begingroup$ but arent these bounds representing the idea of "risk" (just as confidence intervals do)? it can reminds me of bootstrap estimation ... the smallest parameter estimate (we can call this worst case) and the largest parameter estimate (we can call this best case) ... and then show the distribution of all other estimates. so these bounds that you have shown in a way represent the same idea as a confidence interval? $\endgroup$ Jan 4 at 17:20
  • $\begingroup$ I'm not sure what you mean by 'risk' here. Confidence intervals capture random error or sampling error. They are the variation due to only having a sample observed. They shrink as the sample size goes up. Bounds capture systematic error. They do not shrink as sample size goes up. The bounds I have presented above would each also have associated confidence intervals. For an example of this (and more discussion of the distinction between confidence intervals and bounds) see Figure 1 of ncbi.nlm.nih.gov/pmc/articles/PMC6438811 $\endgroup$
    – pzivich
    Jan 4 at 17:33
  • $\begingroup$ a follow up question - are all types of censoring (e.g. left, right, interval) equally as undesirable in survival analysis? stats.stackexchange.com/questions/638373/… $\endgroup$ Feb 3 at 5:38
17
$\begingroup$

From a purely intuitive point of view: Imagine the extreme case where all the cases are censored. Then you couldn't say anything. Now imagine the other extreme, where no one is censored. Then you wouldn't need survival analysis.

Each observation that is censored is one more case where we don't know what will happen, but have to "guess". The different survival analysis methods are different ways of guessing, with different properties and ways they can go wrong, but all are guesses as opposed to knowledge.

"The patient died on Dec 12, 2023" tells us a lot more than "the patient will die some time after this study ends."

$\endgroup$
9
  • 3
    $\begingroup$ +1. The last line says a lot. $\endgroup$ Jan 4 at 11:31
  • $\begingroup$ thank you! in reference to "The patient died on Dec 12, 202 tells us a lot more than "the patient will die some time after this study ends." ... I understand this ... but how does obvious concept "trickle down" and negatively impact the quality of the survival model estimation? $\endgroup$ Jan 4 at 16:44
  • 2
    $\begingroup$ "Now imagine the other extreme, where no one is censored. Then you wouldn't need survival analysis." We followed all participants for 200 years, and they all died. Survival analysis told us nothing other than how groups differed in survival curves, median survival time, etc. $\endgroup$
    – Alexis
    Jan 4 at 16:56
  • 2
    $\begingroup$ PeterFlom "We have other tools" is very commonly the case for analytic approaches in statistics generally. My point is that even with 0% censoring in survival data, survival analysis/event history analysis allows us to estimate differences in when events occur across groups (or by various continuous predictors), and also permits us to make inferences about those estimates. That hardly seems apposite to "wouldn't need survival analysis". $\endgroup$
    – Alexis
    Jan 4 at 19:01
  • 1
    $\begingroup$ @FrankHarrell You seem to interpret my comments as arguing that survival analysis with 0% censoring is not useful? That is 180° opposite of what I wrote! (i.e. I think you and I are in agreement) :) $\endgroup$
    – Alexis
    Feb 3 at 16:43
16
$\begingroup$

Keep it simple. First think of the purpose of estimation. One of the purposes is to estimate absolute survival probabilities. If you had 10,000 patients and no events you estimate S(t)=1.0 with high precision, so no problem even with all observations censored. For estimation of relative effects, consider the exponential distribution which is a special case of Weibull. The variance of the log hazard rate estimate in an exponential distribution is the reciprocal of the number of uncensored observations. So for estimating relative effects censoring is "bad". We need a similar result for accelerated failure time distributions such as what the log-normal survival model uses. The result will be similar to the exponential case, but not quite the same.

$\endgroup$
6
  • $\begingroup$ thank you so much! if we look at the mathematical formulas for the variances of survival estimates (either S(t), H(t) or the beta coeffecients) .... can we see how more censoring results in larger widths? I am currently trying to work on these formulas and post my progress... $\endgroup$ Jan 4 at 16:55
  • $\begingroup$ Not necessarily. With no censoring and large N, var(S(t)) can be nearly zero. $\endgroup$ Jan 5 at 7:31
  • $\begingroup$ did you mean "with more censoring"? With no censoring, I thought its natural to expect smaller values of var(S(t))? $\endgroup$ Jan 5 at 7:52
  • $\begingroup$ Sorry I meant to see with "all censoring". $\endgroup$ Jan 5 at 9:45
  • $\begingroup$ ah ok... all censoring .... $\endgroup$ Jan 5 at 22:52
12
$\begingroup$

Imagine a two-armed clinical trial intended to study a treatment for some long-term condition, but which only follows-up patients for a single day. This is obviously a rather stupid design, as very likely none of the patients will have an event in just 1 day, and you'll learn absolutely nothing about which treatment would have been more effective over a longer term. The follow-up times are just too short, resulting in everybody being censored.

Note that this scenario is really no different from having a longer follow-up period but still seeing mostly censored individuals. In a survival analysis where most individuals are censored, it suggests that your follow-up period may be inappropriately short to effectively study longer-term effects. It would be no different having a 1-year follow-up time for events that occur on a 100-year time scale. The absolute length of follow-up time or number of individuals in the survival analysis isn't what gives you power, it's the number of events, which is related to how long the follow-up is relative to the event timing. Some amount of censored data is expected, but if a very large proportion of the cohort is censored, intuitively you haven't followed them long enough to study your event of interest, and your power will suffer.

That said, since power is determined by the number of events, you can mitigate a high censoring rate somewhat simply by having a larger cohort with a short follow-up time, allowing you to collect more early events by chance alone. This does, however, run the risk of finding effects unique to the early time points, rather than actually looking at the time scale relevant to most individuals - even if the N were so large as to have proper power with a 99% censor rate, I'd be cautious.

$\endgroup$
1
  • $\begingroup$ thank you so much! if we look at the mathematical formulas for the variances of survival estimates (either S(t), H(t) or the beta coeffecients) .... can we see how more censoring results in larger widths? I am currently trying to work on these formulas and post my progress... $\endgroup$ Jan 4 at 16:55
11
$\begingroup$

Other answers correctly explain the main problem with censoring, that is the loss of power because the sample is actually becoming smaller as more patients leave.

However I think there may be another problem even if the sample is large enough to keep the analysis reasonably powerful after a lot of censoring: censoring may not be completely at random. We assume that the remaining patients are a representative sample of the population, but if the probability of patients leaving is somehow related with the condition we are studying, the remaining sample could be biased. And of course, the more censored it became, the more biased it can be.

In other words, for example, if the healthiest patients are more likely to leave the study, and a lot of patients leave, we might end seriously underestimating survival time. Or if the patients in worse condition are more likely to abandon, and a lot of patients abandon, we may end overestimating survival time.

If there is little censoring (very few people leave) our end sample will be very similar to the original sample, and therefore the potential bias will be less than when there is a large amount of censoring.

$\endgroup$
2
  • 1
    $\begingroup$ +1 Relevant: Hernán, M. A. (2010). The Hazards of Hazard Ratios. Epidemiology, 21(1), 13–15. $\endgroup$
    – Alexis
    Jan 4 at 20:39
  • $\begingroup$ Why is that relevant here? $\endgroup$ Feb 3 at 12:30
6
$\begingroup$

Nitpicking point: there are some answers something along the lines of "if all the data was censored, you would not be able to estimate the survival curve".

To be slightly more precise, if all the data were right censored (which is the most common type of censoring), then at best you could estimate an lower bounds of the survival probability at the observed censoring points.

However, there are some cases in which we can get reasonable estimates when 100% of the data is censored. For example, current status data occurs when we inspect each item at one time and observe that the event has not occurred (right censored) or already occurred at the inspection time (left censored). In this case, 100% of the data is censored but we still can get consistent estimates of the survival distribution!

But to drive the point home that uncensored data is more informative, the convergence rate of the Non-Parametric Maximum Likelihood Estimate (NPMLE, generalization of the Kaplan Meier curves) with current status data is $n^{-1/3}$ compared to the more standard convergence rate of $n^{-1/2}$. So this convergence rate reinforces our heuristic understanding that knowing the exact event time should be more informative than knowing that the event occurred before or after a given time.

$\endgroup$
1
  • 3
    $\begingroup$ I don't think it's really that tricky. With 100% censoring but some censoring times being moderately large, you can estimate the survival curve (a constant at 1.0) very well. $\endgroup$ Jan 5 at 7:32
3
$\begingroup$

As an illustrative example, consider estimation of the failure survival curve under independent censoring. Let $T,C>0$ be the potential failure and censoring times, respectively. Define $F(u) = P(T>u)$ and $G(u) = P(C \geq u)$. The Kaplan Meier estimator is an asymptotically efficient nonparametric estimator of $P(T>u)$ and asymptotically follows a Gaussian process with covariance function $$\sigma^2(s, t) = F(s) F(t) \int_{(0,s \wedge t]} \frac{1}{G(u)} \,\mathrm{d} \left\{ \frac{1}{F(u)} \right\}.$$ Notice the $G$ in the denominator: if censoring is more aggressive, then $G$ is smaller, and $\sigma^2$ is larger. In fact, this argument establishes the following result.

Theorem: Let $T>0$ and $C_{\text{hi}},C_{\text{lo}}>0$. Consider observed data under $C_{\text{hi}}$ and under $C_{\text{lo}}$. If $C_{\text{hi}}$ stochastically dominates $C_{\text{lo}}$, then the Kaplan Meier estimator calculated under $C_{\text{hi}}$ has lower variance than the Kaplan Meier estimator calculated under $C_{\text{lo}}$.

$\endgroup$
2
  • $\begingroup$ wow! this is a very theoretical answer! $\endgroup$ Jan 8 at 22:33
  • $\begingroup$ are you a stats prof? I am stuck on some questions (a bountied question on my profile, and a question on statistical assumptions affecting estimation). do you think you can take a look at those please if you have time? $\endgroup$ Jan 8 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.