1
$\begingroup$

I'm looking for a comprehensive guideline of how to compare different survival models on one particular data set. My priority is having a model that gives the best possible prediction of time to event.

By different models here, they can be

  • Models of different types (e.g. CoxPH, parametric PH models, AFT models, random forest, etc.)
  • Models are fitted on different sets of covariates, each set can be mutually exclusive or overlapping with each other, and all sets are subsets of a complete list of covariates
  • Models are fitted on different sets of sample, each comes from a larger pool of samples

I would like to describe the context of my problem if specific details are needed:

  • I'm working on estimating time-to-default of consumer loans
  • My dataset contains about 200,000 loans originated from 2017 to 2020
  • The proportion of censored observation (loans without default) is considerable (approximately 70%)

I need guidelines on these specific topics:

  • Given the nature of the models I'm testing, what can be a good metric to compare them? I found that some metrics depend on sample size and the number of covariates (e.g AIC), and c-index can be problematic regarding highly censored data.
  • If evaluation on a hold out set is needed, is there any rules for choosing it? (The data I'm working on has a great portion of censoring)

I really appreciate if anyone can give me some comments.

Thanks!

$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! I'm a bit unclear on what you mean by: "Models are fitted on different sets of sample." Are you fitting models on different subsets of the same data, different data sets drawn from the same population, or something else? Please edit the question to explain that and provide a concrete example of what's involved; comments are easy to overlook and can be deleted. $\endgroup$
    – EdM
    Jun 10, 2023 at 16:21
  • $\begingroup$ @EdM I apologize for my late reply. Thank you so much for your input. I have edited the post to include (1) the description of sample sets, which all come from one pool of sample, and (2) some description of the dataset (observations are originated from different point of time) $\endgroup$
    – Dalliance
    Jun 13, 2023 at 7:14

2 Answers 2

0
$\begingroup$

When evaluating survival models with different covariate sets and sample sizes, there are several key steps you can follow to ensure a proper evaluation:

  1. Data Preparation: Start by preparing your data for analysis. This includes cleaning the data, handling missing values, and ensuring that your covariates are properly encoded and formatted.
  2. Splitting the Data: Divide your data into two or more subsets for training and evaluation. The most common approach is to split the data into a training set and a separate validation or test set. The training set is used to build the model, while the validation or test set is used to assess its performance. The splitting should be done in a way that preserves the temporal order of the data to reflect the survival nature of the problem.
  3. Model Building: Construct survival models using different covariate sets and sample sizes. You can choose from various survival models such as Cox proportional hazards model, accelerated failure time model, or parametric survival models, depending on your specific requirements. Fit each model to the training data, taking into account the appropriate model assumptions.
  4. Performance Measures: Evaluate the performance of each model using appropriate performance measures. Common evaluation metrics for survival models include concordance index (C-index), Brier score, integrated Brier score, and survival calibration. These metrics assess the model's ability to predict survival outcomes accurately.
  5. Cross-Validation: To reduce the impact of sample variability, consider using cross-validation techniques such as k-fold cross-validation. This involves splitting the data into k subsets, training the models on k-1 subsets, and evaluating their performance on the remaining subset. Repeat this process for different folds and average the results to obtain a more robust assessment of model performance.
  6. Comparison and Selection: Compare the performance of the different models based on the evaluation metrics. Consider the covariate sets and sample sizes that yield the best performance. It is essential to consider not only the overall performance but also the interpretability and practical significance of the covariates included in the models.
  7. External Validation: Once you have identified the best-performing model(s) based on the evaluation, perform external validation if possible. This involves applying the selected model(s) to an independent dataset to assess their generalizability and robustness.

Remember that the evaluation process is iterative, and it may require experimenting with different covariate sets, model specifications, and sample sizes to find the optimal combination that yields the best predictive performance for your specific survival analysis task.

$\endgroup$
0
$\begingroup$

When evaluating survival models, it is important to consider the covariate set and sample size used. Here are some tips to properly evaluate survival models of different covariate set and sample size:

  1. Use appropriate performance metrics: The most common performance metrics for survival models are concordance index (C-index), area under the receiver operating characteristic curve (AUC), and integrated Brier score (IBS). Choose the appropriate metric based on the research question and the characteristics of the data.
  2. Compare models using the same metric: To properly compare survival models of different covariate set and sample size, use the same performance metric. This ensures that the comparison is fair and unbiased.
  3. Use cross-validation: Cross-validation is a method that divides the data into training and validation sets. This ensures that the model is evaluated on data that was not used for training. Use cross-validation to assess the generalizability of the model.
  4. Consider the complexity of the model: When comparing survival models of different covariate set and sample size, consider the complexity of the model. A more complex model may fit the data better, but may not generalize well to new data. Use information criteria, such as Akaike information criterion (AIC) and Bayesian information criterion (BIC), to balance between model complexity and goodness of fit.
  5. Interpret the results: When evaluating survival models of different covariate set and sample size, interpret the results in the context of the research question and the characteristics of the data. Consider the practical significance of the results, not just the statistical significance.

By following these tips, you can properly evaluate survival models of different covariate set and sample size and make informed decisions based on the results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.