I am working with the standard proportional hazards model given by $\lambda(t|Z) = \lambda_0(t)e^{Z\beta}$ for a special type of data that requires an EM algorithm to estimate a discretized version of $\lambda_0$, and $\beta$. I have another data set consisting of right-censored failure time data that I want to include in this EM algorithm procedure. I am aware that the partial likelihood is the standard procedure however I cannot use this approach I am constrained to using an EM algorithm.


Can an EM algorithm be applied to right-censored failure time under a proportional hazards model to estimate the unknown parameters? If so, are there any references which cite this approach and if not, what would be the justification against its use?

Any help with this problem would be greatly appreciated. Thank you.


Thank you to @EdM. I am trying to use the following algorithm. Let $$(X_i, \delta_i, Z_i): i = 1, 2, ..., n$$ be right-censored data with covariates Z. Defining the "missing" data as those failure times that are censored, the expected log-likelihood function is given by: $$l_{E}(\beta, \lambda) = \sum_{i=1}^{n} \sum_{j=1}^{k} w_{ij} \log(f_i(t_j))$$ where $w_{ij} = \mathbb{E}(1(T_i = t_j) | O) = \delta_i 1(X_i = t_j) + (1-\delta_i) \frac{p_{ij}1_(X_i \leq t_j)}{\sum_{j=1}^{k} p_{ij}1(X_i \leq t_j)}$ with $$p_{ij} = \lambda_j \exp(\beta Z_i) \exp( \sum_{l=1}^{j} \lambda_l \exp(\beta Z_i))$$ and $$f_{i}(t_j) = \lambda_j \exp(\beta Z_i) \exp( \sum_{l=1}^{j} \lambda_l \exp(\beta Z_i))$$ The algorithm would proceed by computing the $w_{ij}$ weights, maximizing the function with respect to the baseline hazards $\lambda_j$ and coefficients $\beta$ and then iterating until convergence. I assumed the last observed time is a failure so no probability mass was lost in the tail but the algorithm does not appear to be working. I also cannot find any mention of this approach in the literature.

  • $\begingroup$ To clarify: you are trying to estimate the baseline hazard function at each event time $t_j$, as well as the regression coefficients. Is that correct? Also, be careful with the assumption that the last observed time is a failure. $\endgroup$ – EdM May 26 at 16:53
  • $\begingroup$ Yes, that is correct. I am simulating my data at this point, so I can control for this assumption. It seems this setup is similar to a discrete proportional hazards model with an EM algorithm...but yet again there is very little literature $\endgroup$ – J McVittie May 26 at 16:57
  • $\begingroup$ Are there some assumptions underlying the EM algorithm that break for right-censored data or the proportional hazards model that breaks under the EM algorithm? $\endgroup$ – J McVittie May 27 at 12:07
  • $\begingroup$ I'm not aware of anything that necessarily "breaks" in combining EM with right censoring and PH. If your attempt is failing, please edit the question to be more specific about the nature of the failure. While I'm thinking about the EM issue (not one of my strengths), please say more about why you need to use EM. Is there some reason why a smoothed baseline hazard estimate (e.g., via penalized regression) won't work? Are you also trying to estimate time-discrete/time-dependent values of the $\beta$s? $\endgroup$ – EdM May 27 at 13:43
  • $\begingroup$ Thanks, I looked at my code more carefully and found a typo. I think the algorithm is fine, its simply a coding issue. Thanks again for all your help! $\endgroup$ – J McVittie May 27 at 15:05

EM algorithms have long been used in survival modeling in situations where EM is useful. For example, some survival frailty analysis uses EM algorithms. Hougaard covers that in the 2000 book Analysis of Multivariate Survival Data, and the R frailtyEM package implements an EM approach to frailty modeling.

The ICsurv package uses EM to handle the problems introduced by interval censoring in survival models. The smcure package uses EM to fit cure models.

So in principle there is no problem with doing this type of survival analysis with EM. The specific details in your situation aren't completely clear from the question, but the code for the above packages should provides a roadmap to implementing it.

  • $\begingroup$ @JMcVittie the comment formatting is a bit mangled. Why don't you edit the question to include the specifics? (I'm afraid I won't be of much help in implementation per se, however.) $\endgroup$ – EdM May 26 at 15:31
  • $\begingroup$ Thank you. I have edited to question so that it is more specific. $\endgroup$ – J McVittie May 26 at 15:40

The algorithm does function correctly when coded correctly. Although this is not an computationally optimal approach for modelling the regression coefficients, it does allow one to estimate both the baseline hazard function non-parametrically and the regression coefficients through a single algorithm.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.