7
$\begingroup$

I am told that one of the main benefits of Survival Analysis models are their ability to handle Censored Data. This is in contrast to standard regression models that are unable to do so.

For example, suppose researchers in a medical study are interested in knowing if a certain drug is able to prolong the life of patients with a certain disease : in this medical study, a patient dying is considered as the "event. Suppose one of the patients has to move to a new country 5 years after the study has started and we are no longer able to collect data on this patient. We know that the patient survived for at least 5 years. In a classical regression model, the data for this patient would be considered as "incomplete" and we would not be able to use the data for this patient in the model. However, a Survival Analysis model would let us use to the "complete part of the incomplete data" belonging to this patient - thus allowing our model to profit from potentially useful information that would have been otherwise discarded by a standard regression model. In the context of Survival Analysis, this particular patient would be labelled as "Censored".

I am interested in the following (obvious) question: Suppose we have a dataset that contains no Censored Data - for the purpose of a simulation, we decide to randomly "censor" the data belonging to some of the patients. Can we somehow show that estimates from Survival Models (e.g. Kaplan-Meier, AFT, Cox PH) would be "better" on the same dataset when there is less censoring compared to more censoring? (e.g. one dataset has no censoring, one dataset has 5% random censoring, one dataset has 10% random censoring - we fit Survival Models to all 3 datasets and compare the quality of the estimates)

I am aware that Survival Models do not require there to be Censoring in the dataset, and I am also aware that higher levels of Censoring are considered undesirable for Survival Analysis models - but is there some mathematical proof that shows the "decline" in the estimates provided by Survival Analysis models when higher levels of Censoring are present?

$\endgroup$

2 Answers 2

7
$\begingroup$

We could rephrase your question asking whether methods based on full data (i.e. noncensored data) are necessarily more efficient than methods based on observed data (i.e. censored data). This question can be answered in general by semiparametric efficiency theory.

Let $Z$ denote the full data (such as covariates and failure time). Suppose we have a data set of i.i.d. draws $Z_1, \dots Z_n$. A full data estimator $\hat\beta$ for an estimand $\beta^*$ is asymptotically linear with influence function $\varphi^F$ if $$\sqrt{n} ( \hat\beta - \beta^*) = \frac{1}{\sqrt{n}} \sum_{i=1}^n \varphi^F(Z_i) + o_P(n^{-1/2}).$$ Such an estimator has asymptotic variance $\mathrm{var}\left\{ \varphi^F(Z) \right\}$. Likewise, let $\mathcal{O}$ be the observed data, which denotes the full data $Z$ subject to coarsening or missingness. We can similarly define the influence function $\varphi$ for an observed data estimator.

This suggests that we can compare the efficiency of observed data estimators and full data estimators through comparisons of their influence functions. Rather than studying the influence function of a given estimator, we can study the class of influence functions of all regular estimators of the estimand $\beta^*$.

Lemma 7.4 in Tsiatis (2006) establishes the relationship between the class of influence functions of observed data estimators and the corresponding class for full data estimators. He shows that the class of observed data influence functions equals \begin{equation*} \frac{I(\mathcal{C}=\infty)}{\varpi(\infty, Z)} \varphi^F(Z) + L_2(\mathcal{O}), \end{equation*} where $\mathcal{C}=\infty$ denotes that the full data is observed ( i.e. $T \leq C$ in survival analysis), $\varpi(\infty, Z) = \mathbb{P}[\mathcal{C}=\infty \mid Z]$ is the conditional probability of observing the full data $L_2$ is an arbitrary function satisfying $\mathbb{E}[L_2(\mathcal{O})\mid Z] = 0$, and $\varphi^F$ is an arbitrary full data influence function.

Based on this identity, we can derive the asymptotic variance of an observed data asymptotically linear estimator with influence function $\varphi$ as \begin{align*} & \mathrm{var} \left\{ \varphi(\mathcal{O}) \right\} \\ =\, & \mathrm{var} \left[ \mathbb{E} \left\{ \varphi(\mathcal{O}) \mid Z \right\} \right] + \mathbb{E} \left[ \mathrm{var} \left\{ \varphi(\mathcal{O}) \mid Z \right\} \right] \\ =\, & \mathrm{var} \left[ \mathbb{E} \left\{ \frac{I(\mathcal{C}=\infty)}{\varpi(\infty, Z)} \varphi^F(Z) + L_2(\mathcal{O}) \mid Z \right\} \right] + \mathbb{E} \left[ \mathrm{var} \left\{ \varphi(\mathcal{O}) \mid Z \right\} \right] \\ =\, & \mathrm{var} \left[ \mathbb{E} \left\{ \frac{I(\mathcal{C}=\infty)}{\varpi(\infty, Z)} \varphi^F(Z) \mid Z \right\} \right] + \mathbb{E} \left[ \mathrm{var} \left\{ \varphi(\mathcal{O}) \mid Z \right\} \right] \\ =\, & \mathrm{var} \left[ \varphi^F(Z) \right] + \mathbb{E} \left[ \mathrm{var} \left\{ \varphi(\mathcal{O}) \mid Z \right\} \right] \\ \succcurlyeq\, & \mathrm{var} \left[ \varphi^F(Z) \right] & \end{align*}

This shows that any observed data estimator has higher variance than its corresonding full data estimator. The inequality is tight when the second summand has conditional variance zero: this means that the observed data equals the full data. In a survival analysis setting, this shows that whenever censoring is present, the observed data estimators are less efficient than the full data estimators.

$\endgroup$
6
  • 5
    $\begingroup$ I understand this answer requires technical background and so may miss the mark of what the OP desired. However, I worked it out because it was interesting to me and may be to others as well. $\endgroup$
    – Ben
    May 15 at 17:35
  • $\begingroup$ @ Ben: Thank you so much for your answer! I will spend some time trying to make sense of the mathematics! I think I understand the general idea you have (elegantly) present: estimators with censored data will necessarily have higher variance (i.e. high variance in this case is bad) than estimators with uncensored data. $\endgroup$
    – stats_noob
    May 15 at 17:44
  • 1
    $\begingroup$ @stats_noob That's exactly right: if I gave you an estimator based on censored data, you could give me back an estimator based on uncensored data which has lower variance. $\endgroup$
    – Ben
    May 15 at 17:45
  • $\begingroup$ Have you worked with Survival Analysis before? I am trying to learn more and teach myself about this field. Lately, I have been trying to understand about some specific applications of Survival Analysis - in particular, this question here: stats.stackexchange.com/questions/552569/… . Do you have any comments on this? $\endgroup$
    – stats_noob
    May 15 at 17:45
  • $\begingroup$ @ Ben : Thank you so much for your reply! $\endgroup$
    – stats_noob
    May 15 at 17:55
6
$\begingroup$

This comes down to the central role of events in survival models. Consider, for example, the Nelson-Aalen non-parametric estimate of a cumulative hazard function $\Lambda(t)$ and the variance of the estimate (Equations 2.2 and 2.4 of Therneau and Grambsch):

$$ \hat \Lambda(t) = \sum_{i:t_i\le t}\frac{\Delta \bar N (t_i)}{\bar Y(t_i)}$$

$$\text{var}[\hat \Lambda(t)] = \sum_{i:t_i\le t}\frac{\Delta \bar N (t_i)}{\bar Y^2(t_i)}$$

where $i$ indexes event times, $\Delta \bar N(t_i)$ is the increase in event numbers at time $t_i$, and $\bar Y(t_i)$ is the number at risk at time $t_i$.

Right censoring an event time that would have been at $t_j$ removes information about the change in cumulative hazard that should have been seen at time $t_j$. It also lowers the number at risk at times between its censoring time and $t_j$, increasing the variance of the curve calculated from the remaining events. Censoring event times thus harms survival-model estimates in two ways.

This page discusses how adding censored cases doesn't contribute to power in Cox models. For parametric models, a known event time provides a contribution to likelihood proportional to the probability density of an event at that specific time. A right-censored event time only provides a term proportional to the survival function up to the censoring time. This page shows the formulas.

$\endgroup$
9
  • $\begingroup$ Thank you for your answer! In general, it's fair to state that data with less censoring is more advantageous than data with more censoring? $\endgroup$
    – stats_noob
    May 15 at 15:35
  • $\begingroup$ stats.stackexchange.com/questions/273731/… $\endgroup$
    – stats_noob
    May 15 at 15:35
  • $\begingroup$ Do you have an opinion on this? stats.stackexchange.com/questions/552569/… $\endgroup$
    – stats_noob
    May 15 at 15:38
  • 1
    $\begingroup$ @stats_noob for your first comments, it's (almost) all about the number of events. Less censoring at constant sample size means more events, hence better. With respect to your other question, I recall up-voting the answer from Todd D when it was first posted last fall. The newer answers provide different perspectives. Choice of therapy for an individual might be based on clinical standards derived from survival analysis, but not resource allocation among patients as that question seems to suggest. $\endgroup$
    – EdM
    May 15 at 15:49
  • $\begingroup$ @ EdM : thank you for your reply. I have been spending a lot of time reading about this and trying to teach myself more about this. Just for arguments sake - suppose some medical experts are able to determine that variability in disease progression is primarily dependent on whether the patient is male or female. If we make survival curves, is it completely unreasonable (from a statistics standpoint) to say that "males are more at risk at some times compared to females", and their risk should be taken more seriously at those times? Should this be done with Hazard Curves instead? $\endgroup$
    – stats_noob
    May 15 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.