How to properly evaluate survival models of different covariate set and sample size

Question

I'm looking for a comprehensive guideline of how to compare different survival models on one particular data set. My priority is having a model that gives the best possible prediction of time to event.

By different models here, they can be

• Models of different types (e.g. CoxPH, parametric PH models, AFT models, random forest, etc.)
• Models are fitted on different sets of covariates, each set can be mutually exclusive or overlapping with each other, and all sets are subsets of a complete list of covariates
• Models are fitted on different sets of sample, each comes from a larger pool of samples

I would like to describe the context of my problem if specific details are needed:

• I'm working on estimating time-to-default of consumer loans
• My dataset contains about 200,000 loans originated from 2017 to 2020
• The proportion of censored observation (loans without default) is considerable (approximately 70%)

I need guidelines on these specific topics:

• Given the nature of the models I'm testing, what can be a good metric to compare them? I found that some metrics depend on sample size and the number of covariates (e.g AIC), and c-index can be problematic regarding highly censored data.
• If evaluation on a hold out set is needed, is there any rules for choosing it? (The data I'm working on has a great portion of censoring)

I really appreciate if anyone can give me some comments.

Thanks!

Answer 1

When evaluating survival models with different covariate sets and sample sizes, there are several key steps you can follow to ensure a proper evaluation:

1. Data Preparation: Start by preparing your data for analysis. This includes cleaning the data, handling missing values, and ensuring that your covariates are properly encoded and formatted.
2. Splitting the Data: Divide your data into two or more subsets for training and evaluation. The most common approach is to split the data into a training set and a separate validation or test set. The training set is used to build the model, while the validation or test set is used to assess its performance. The splitting should be done in a way that preserves the temporal order of the data to reflect the survival nature of the problem.
3. Model Building: Construct survival models using different covariate sets and sample sizes. You can choose from various survival models such as Cox proportional hazards model, accelerated failure time model, or parametric survival models, depending on your specific requirements. Fit each model to the training data, taking into account the appropriate model assumptions.
4. Performance Measures: Evaluate the performance of each model using appropriate performance measures. Common evaluation metrics for survival models include concordance index (C-index), Brier score, integrated Brier score, and survival calibration. These metrics assess the model's ability to predict survival outcomes accurately.
5. Cross-Validation: To reduce the impact of sample variability, consider using cross-validation techniques such as k-fold cross-validation. This involves splitting the data into k subsets, training the models on k-1 subsets, and evaluating their performance on the remaining subset. Repeat this process for different folds and average the results to obtain a more robust assessment of model performance.
6. Comparison and Selection: Compare the performance of the different models based on the evaluation metrics. Consider the covariate sets and sample sizes that yield the best performance. It is essential to consider not only the overall performance but also the interpretability and practical significance of the covariates included in the models.
7. External Validation: Once you have identified the best-performing model(s) based on the evaluation, perform external validation if possible. This involves applying the selected model(s) to an independent dataset to assess their generalizability and robustness.

Remember that the evaluation process is iterative, and it may require experimenting with different covariate sets, model specifications, and sample sizes to find the optimal combination that yields the best predictive performance for your specific survival analysis task.

Answer 2

When evaluating survival models, it is important to consider the covariate set and sample size used. Here are some tips to properly evaluate survival models of different covariate set and sample size:

1. Use appropriate performance metrics: The most common performance metrics for survival models are concordance index (C-index), area under the receiver operating characteristic curve (AUC), and integrated Brier score (IBS). Choose the appropriate metric based on the research question and the characteristics of the data.
2. Compare models using the same metric: To properly compare survival models of different covariate set and sample size, use the same performance metric. This ensures that the comparison is fair and unbiased.
3. Use cross-validation: Cross-validation is a method that divides the data into training and validation sets. This ensures that the model is evaluated on data that was not used for training. Use cross-validation to assess the generalizability of the model.
4. Consider the complexity of the model: When comparing survival models of different covariate set and sample size, consider the complexity of the model. A more complex model may fit the data better, but may not generalize well to new data. Use information criteria, such as Akaike information criterion (AIC) and Bayesian information criterion (BIC), to balance between model complexity and goodness of fit.
5. Interpret the results: When evaluating survival models of different covariate set and sample size, interpret the results in the context of the research question and the characteristics of the data. Consider the practical significance of the results, not just the statistical significance.

By following these tips, you can properly evaluate survival models of different covariate set and sample size and make informed decisions based on the results.