4
$\begingroup$

I am wondering if there's any implementation or theory about using stochastic gradient descent for estimation parameters in Cox proportional hazards model. Normally in that model the partial-log-likelihood (which for right censoring that is independent and uninformative about parameters is a true log-likelihood)

$$\ell_p(\beta) = \sum\limits_{i=1}^{K}X_i'\beta - \sum\limits_{i=1}^{K}\log\Big(\sum\limits_{l\in \mathscr{R}(t_i)}^{}e^{X_l'\beta}\Big) $$

where

  • $K$ denotes number of failure times,
  • $\mathscr{R}(t_i)$ is a risk set at time $t_i$

is estimated using the first and second derivatives and Newton-Raphson algorithm.

I was wondering that if the first derivative can be formulated as a sum of elements in which every element corresponds only to information from 1 observation, then the estimation with the stochastic gradient descent would be easy, but I can't transform this derivative to such form:

$$U_k(\beta)=\dfrac{\partial\ell_k(\beta)}{\partial\beta_k}=\sum\limits_{i=1}^{K}\Big(X_{ik}-A_{ik}\Big), A_{ik} = \dfrac{\sum X_{lk} e^{X_l'\beta}}{\sum e^{X_l'\beta}}$$ and the sums in $A_{ik}$ are over risk set.

Does anyone heard about any articles applying SGD to Cox proportional hazards model? It might have great applications in large-scale cox models.

$\endgroup$
3
$\begingroup$

Mathematicians and computer scientists often think the idea of (mini-batch) SGD is to randomly subsample the terms in your objective function. But from a statistical point of view, SGD is about subsampling the data you are learning from, before you even bother writing down any particular likelihood function. If you want an SGD-type method for Cox regression, you will have to subsample the observed data and construct the objective function from the subsampled data, not subsample the objective function's terms.

For many likelihood models, these views are equivalent, but for Cox they are not. Under a given hazard model for your sample, the partial likelihood function tells you the probability that the events you observed, would be observed in the exact order that they appeared. Unlike most likelihood functions, this likelihood depends on all observations in the set; you can't calculate it separately for subjects 1-5 and 6-10 and then multiply them together. That is why you are finding it impossible to subsample the objective function's terms.

In contrast, it is trivial to do SGD in the data subsampling style, no matter what the likelihood. Simply form a random subsample of the data, construct the objective function for this subsample, then iteratively repeat this procedure with a different random subsample each time.

I can't prove theoretically that this will work, but I'm quite confident it will. Cox regression is very closely related to GLMs, and SGD certainly works for GLMs. It should also help that Cox regression is a convex optimization problem.

$\endgroup$
  • $\begingroup$ Thanks for your comment. Do you rather suggest to reorder samples and then go one by one to estimate coefficients of the cox model by using previously observed samples? So after reordering, for an n-th observation I calculate only 1 factor from the score function based on n-1 previously observed samples and that n-th one AND then actualize coefficients? Or rather I reorder samples, and then go by let's say 5 sample and I only use 5 samples to calculate coefficents based on those observed 5 examples and whole factors from the score function that appear to exist for those 5 observations? $\endgroup$ – Marcin Kosiński Sep 8 '15 at 20:18
  • $\begingroup$ No, I meant use mini-batch SGD but form the minibatches by randomly subsampling from your data at each iteration. At each minibatch iteration, randomly subsample from your data, form the Cox likelihood for the subsample, and perform a gradient descent step using that likelihood. $\endgroup$ – Paul Sep 9 '15 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.