I have a data set containing various subsets of medical data about a cohort of patients. For example there are blood test results, demographics, medical examination results and a medical history among other things.

I have created survival models on each subset of the data individually using various methods such as CoxPH, Random Survival Forests, CoxBoost and penalized Cox Regression. I am getting quite good values for the concordance index in the data subsets, but I would like to build a model that combines all of the individual models, in an attempt to improve accuracy.

One option is to select the most important features from the individual data subsets and combine those and build a survival model from those. This does improve accuracy. But is there a way to combine the results of the individual survival models themselves, perhaps by simply aggregating their results in some way like majority voting or by having a higher level algorithm make a decision based on the decisions of the individual survival models? Would this be valid in the case of the Cox PH model for example? What are the issues to consider?

  • $\begingroup$ What do you mean by "each subset of the data"? Do you have one large cohort on which you have done separate survival analyses, e.g.: one based on blood tests, one based on demographics, one based on medical examination results, and one based on medical history? Or do you have different cohorts each analyzed for each of these types of data? If the latter, how much overlap is there among the cohorts? $\endgroup$ – EdM Mar 28 at 14:56
  • $\begingroup$ I have one cohort of 873 patients and I have done separate survival analyses on the entire cohort, one for blood tests, one for demographics and so on as you suggested. The reason for this is the large number of variables in each of these subsets. So I am wondering if it is possible to combine these results in some way. $\endgroup$ – panda Mar 29 at 1:14

It is possible to combine the prediction from several survival models by using techniques from ensemble learning. In fact, a Random Survival Forest is already an ensemble of survival trees whose predictions are aggregated to form the final prediction of the entire forest. You can use a similar approach when combining predictions from different survival models, which would be called an heterogeneous ensemble.

If you have models that just output a single risk score, which would be the case for all Cox-like models, you just have to aggregate a list of numbers. The simplest way is to just compute an average; or, you can train an additional (linear) survival model on top of the risk scores to learn how to fuse individual predictions – this is called stacking. Even more sophisticated you could consider pruning models that either perform poorly or whose predictions are strongly correlated – this is called ensemble selection or ensemble pruning. For an example for the latter, you can checkout the paper Heterogeneous ensembles for predicting survival of metastatic, castrate-resistant prostate cancer patients .

The most important consideration for an ensemble to be better than its individual members is that their predictions are uncorrelated. If predictions are strongly correlated, all models would essentially give you the same answer, thus, aggregation would not give a different answer. In fact, uncorrelated predictions are usually more important than the prediction performance of individual models, as long as individual models perform better than random (concordance index > 0.5).

  • $\begingroup$ Thanks for the paper reference - that is very helpful, although not exactly what I am trying to do as each of my models is run only on a subset of the features available. $\endgroup$ – panda Mar 29 at 1:17

Your concern about having too many predictors for the number of cases is valid, but there are better ways to proceed than trying to combine models based on separate subsets of predictors within the same cohort.

The problem is that survival models have an inherent omitted-variable bias. Unlike linear regression, where this is only a problem if the predictors omitted are correlated with those included, omitting any predictor associated with outcome from a survival or logistic regression model tends to bias the estimates of all the included predictor coefficients toward lower magnitudes. This answer has a nice analytical explanation for a conceptually similar situation with probit models.

So the problem you face is trading off that omitted-variable bias against the danger of overfitting with too many predictors.

Say that you have 450 events among your 873 patients. Then you could reasonably try to include 30 or 40 predictors in a standard unpenalized survival model without overfitting. So one approach would be to use clinical judgment to identify the 30 or 40 predictors among all the data types that are most likely to be related to outcome, and use them in a standard Cox regression.

But you already are using penalized approaches in some of your modeling attempts. Your "penalized Cox regression" is presumably a ridge regression or LASSO, which cuts down the effective number of predictors and adjusts the coefficient values to reduce overfitting. A ridge regression using all of the predictors might work surprisingly well; there's no reason to do ridge on separate subsets of them.

Random forests also tend to minimize the possibility of overfitting as they don't use all of the predictors at each decision point. Boosted trees don't typically overfit unless you use too many trees. These tree-based methods are essentially ensembles of models to start with.

Different types of models have different strengths, and there might be something to be said for combining them. So you could consider the following approach: develop different types of models based on all of the predictors reasonably associated with outcome and then combine the models in some way. For example, Mark van der Laan and colleagues recommend what they call targeted maximum likelihood estimation in which they build large numbers of different types of models and then develop a parametric model combining the models' predictions near a particular point of interest (e.g., survival at 3 years) that is optimized by maximum likelihood techniques. That combination of information among several models can lessen any overfitting provided by any single model.

That said, you could be just as well off in practice with one of a well chosen standard Cox model, ridge regression, or one of the tree-based approaches. Penalized approaches like ridge regression have the advantage that you can easily choose to include some critical predictors without penalization and include all other predictors with penalization.

One important point here is to get away from analyzing separate subsets of predictors and to include together as many predictors as are reasonably related to outcome in the set of predictors that you evaluate for your model building. A second point, critical but not directly raised in your your question, is to evaluate your model building process with methods like bootstrapping to get an estimate of generalizability and bias. This page has an outline and links to further details.

  • $\begingroup$ Thanks for your comprehensive answer. I will read all the refs you have provided. The data is about 90% censored - I have only 82 events. I have over 1000 predictors. I have used some of my models (Lasso, Random Survival Forests, CoxBoost) to select the most important predictors from each subset of data and put these into a CoxPH model. It gives a higher c-index than the individual data subsets but maybe from what you are saying that is not entirely valid either. I have been using 5-fold CV but plan to use Stability Selection with subsampling to improve the stability of my feature selection. $\endgroup$ – panda Mar 29 at 4:03
  • $\begingroup$ @panda pre-selecting predictors and then using them in a standard regression runs a risk that the p-values will no longer be valid and that your model would not apply well to a new sample from the population. See this answer and its comments for the problems and links to potential ways around them. If you could automate your entire model-bulding and variable-selection process, bootstrapping as described in the last link in my answer could provide information about how well your process works. $\endgroup$ – EdM Mar 29 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.