How does Censoring in Survival Analysis impact Confidence Intervals?

Question

I am trying to understand why large amounts of censoring is unfavorable in survival analysis. Logically, this makes sense (ex : Why is large amounts of censoring bad in survival analysis? , in the answers "The patient died on Dec 12, 2023" tells us a lot more than "the patient will die some time after this study ends." )

But I am trying to see if there something in the mathematical/statistical formulas used in survival estimation can be used to justify why more censoring is unfavorable compared to less censoring (ex: perhaps more censoring results in larger variance estimates for the regression coefficients, survival function, hazard function)

Non Parametric Approach: I tried to do this myself for the Kaplan-Meier (non-parametric) approach as this is probably easier compared to the full parametric and semi-parametric approaches.

Here is the Kaplan-Meier estimator for the Survival Function and the corresponding Variance (here, $d_i$ is the number of events at time $t_i$ and $n_i$ is the number of medical patients at risk at time $t_i$):

$$ \hat{S}(t) = \prod_{i: t_i \leq t} \left(1 - \frac{d_i}{n_i}\right) $$ $$ Var[\hat{S}(t)] = \hat{S}(t)^2 \sum_{i: t_i \leq t} \frac{d_i}{n_i(n_i - d_i)} $$

I thought of the two following cases : Case 1 has some censoring and Case 2 has no censoring. I will compare the variances of both cases to try and if Case 2 results in better variances compared to Case 1. This will allow me to see the impact of censoring on variance estimates (ex: in both cases, t1=t1, t2=t2 and t4=t4):

Case 1 (Some Censoring):

• patient1 has event at t1
• patient2 has event at t2
• patient3 drops out of the study at t3
• patient4 has event at t4
• and when the study is over at t5, patient 5 has not had the event

Case 2 (No Censoring):

• patient1 has event at t1
• patient2 has event at t2
• patient3 has event at t3
• patient4 has event at t4
• patient5 has event at t5

Here is my attempt for the variance calculations for both cases:

Case 1:

• At time $t_1$, $n_1=5$, $d_1=1$. The variance is $\hat{S}(t_1)^2 \frac{1}{5(5 - 1)}$.
• At time $t_2$, $n_2=4$, $d_2=1$. The variance is $\hat{S}(t_2)^2 \frac{1}{4(4 - 1)}$.
• At time $t_3$, $n_3=3$, $d_3=0$. The variance is $0$ (since $d_3=0$).
• At time $t_4$, $n_4=2$, $d_4=1$. The variance is $\hat{S}(t_4)^2 \frac{1}{2(2 - 1)}$.
• At time $t_5$, $n_5=1$, $d_5=0$. The variance is $0$ (since $d_5=0$).

The total variance in Case 1 is the sum of these variances:

$$ Var_{total, Case1} = \hat{S}(t_1)^2 \frac{1}{5(5 - 1)} + \hat{S}(t_2)^2 \frac{1}{4(4 - 1)} + 0 + \hat{S}(t_4)^2 \frac{1}{2(2 - 1)} + 0 $$

Case 2:

• At time $t_1$, $n_1=5$, $d_1=1$. The variance is $\hat{S}(t_1)^2 \frac{1}{5(5 - 1)}$.
• At time $t_2$, $n_2=4$, $d_2=1$. The variance is $\hat{S}(t_2)^2 \frac{1}{4(4 - 1)}$.
• At time $t_3$, $n_3=3$, $d_3=1$. The variance is $\hat{S}(t_3)^2 \frac{1}{3(3 - 1)}$.
• At time $t_4$, $n_4=2$, $d_4=1$. The variance is $\hat{S}(t_4)^2 \frac{1}{2(2 - 1)}$.
• At time $t_5$, $n_5=1$, $d_5=1$. The variance is $\hat{S}(t_5)^2 \frac{1}{1(1 - 1)}$.

The total variance in Case 2 is the sum of these variances:

$$ Var_{total, Case2} = \hat{S}(t_1)^2 \frac{1}{5(5 - 1)} + \hat{S}(t_2)^2 \frac{1}{4(4 - 1)} + \hat{S}(t_3)^2 \frac{1}{3(3 - 1)} + \hat{S}(t_4)^2 \frac{1}{2(2 - 1)} + \hat{S}(t_5)^2 \frac{1}{1(1 - 1)} $$

If we look at the total variance, it seems (contradictorily) that Case 2 (without censoring) has larger total variance compared to Case 1 (with censoring)?

Parametric Approach (AFT):

Here is the model and likelihood for a parametric survival model (AFT)

$$ \log(T) = \mu + \beta^T X + \sigma \epsilon $$

$$ L(\mu, \sigma, \beta) = \prod_{i=1}^{n} \left[ f\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{\delta_i} \left[ 1 - F\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{1-\delta_i} $$

Note that for the likelihood function in a parametric survival model, non-censored patients contribute to the likelihood via their exact time of event (first term in the likelihood) and censored patients contribute to the likelihood via their survival time (second term in the likelihood). This is because we know the exact time of event for non-censored patients and only the survival time of censored patients

Case 1 (full censoring): Imagine a case where all observations are censored, then the likelihood function would only be $L(\mu, \sigma, \beta) = \prod_{i=1}^{n} \left[ 1 - F\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{1-\delta_i}$

Case 2 (no censoring): Imagine a case where all observations are uncensored, then the likelihood function would only be $ L(\mu, \sigma, \beta) = \prod_{i=1}^{n} \left[ f\left( \frac{\log(t_i) - \mu - \beta^T X_i}{\sigma} \right) \right]^{\delta_i} $

Since variance estimates are proportional to 1/Information Matrix, and the Information Matrix is related to the second derivative - I think maybe we could show that the second derivatives will be smaller in Case 1 compared to Case 2 ... thus the variance estimates with full censoring will be larger in Case 2 and Case 2 will have larger Confidence Intervals (unfavorable) compared to Case 1.

Then, I might be able to show that for all intermediate cases that exist between Case 1 and Case 2 (i.e. 1% censoring, 2% censoring .... 99% censoring, etc.) - in general, the higher the proportion of censoring and the more that the likelihood function shifts towards Case 1 ... the more we can expect the second derivatives to be smaller, the variance to be larger and larger confidence intervals. But I am not sure if this form of thinking is correct.

Is this analysis correct? Does censoring have an impact on size of variance estimates? Will this relationship be more visible for AFT and Cox-PH models? Or is total variance generally not evaluated in survival analysis?

Answer 1

What's more important than the censoring fraction is the nature of the censoring.

For example, say that your study evaluated all individuals for some event over a period of 3 years, then stopped so that those who hadn't yet developed the disease had censored event times. If the 3-year event-free survival was 0.8 you might have 80% censoring overall but really good estimates of the event time course over the first 3 years.

Alternatively, say that you tried to set up such a study with the same number of individuals but lost contact with many of them just after the study started. You might still end up with 80% censoring overall, but your survival estimates over the first 3 years would be much worse.

Maybe more important, high censoring fractions in some situations might suggest that censoring is informative about the risk of an event.

For background, I'd recommend that you read the 1997 review by Leung et al., "Censoring Issues in Survival Analysis," in Annu. Rev. Public Health 18: 83-104.

In terms of your particular examples, the setup of the likelihoods for the accelerated failure time (AFT) model seems OK. Yes, an observed event time provides more information than a right-censored event time. The question you might want to address is how the pattern, not just the fraction, of censored event times affects the coefficient estimates. That would best be done by simulation.

The formulas for the Kaplan-Meier model are correct, but the implementation in your Cases 1 and 2 is incorrect. In particular, in:

$$ Var[\hat{S}(t)] = \hat{S}(t)^2 \sum_{i: t_i \leq t} \frac{d_i}{n_i(n_i - d_i)} $$

note that (1) there is a factor of $\hat{S}(t)^2$ whose actual values weren't used and (2) the sum is over all event times up to the time of interest, while your calculations just used the numbers at the specific time. For example, in your Case 1 at time $t_4$ (Kaplan-Meier survival estimate of 0.3), the variance of the estimate is:

$$0.3^2 \left(\frac{1}{5\times 4} + \frac{1}{4\times 3} + \frac{1}{2\times 1}\right)= 0.057 .$$

Even after those matters are fixed, it makes no sense just to add up the variances at the event times to get an overall estimate of the variance. If there are fewer event times there will necessarily be fewer terms to add up. What's usually of more interest is the precision of survival estimates from the model. It's usually the number of events, not the total number of cases or the censoring fraction, that determines that precision.