12
$\begingroup$

I used a regularized (LASSO) Cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model.

Question: Could I use the same method to correct the regression coefficients of my best model (based on minimum lambda)?

optimism-corrected b = b of best model - optimism estimated by Harrell's method for prediction accuracy

Would be such an optimism-corrected b be a better predictor for unseen cases?

$\endgroup$
3
  • 1
    $\begingroup$ Got a reference to Harrell's publication on this correction? $\endgroup$ Mar 19, 2017 at 19:33
  • 1
    $\begingroup$ Would you mind providing links? I don't have enough information in the question to be able to engage it. $\endgroup$ Jan 30, 2019 at 19:10
  • $\begingroup$ I assume the easiest and shortest description of Harrell's method is link but the original paper by Harrell et al 1996 provides more information link. I came to the conclusion not to do it but to recalibrate my model using optimism corrected callibration slopes. $\endgroup$ Mar 7, 2019 at 12:43

1 Answer 1

2
$\begingroup$

This goes against what Harrell seems to mean when he writes about bootstrap validation.

Harrell's argument basically goes like this.

  1. Splitting the data wastes data that could have been used for training, so train on the entire dataset.

  2. However, then we risk overfitting. We always risk overfitting, but when we have holdout data, we can catch that we have just played connect-the-dots.

  3. Therefore, bootstrap your data, fit the model to the bootstrap sample, evaluate that model on the full data set, and see how that performance compares to the performance of the model that was trained on the entire data set. Because we do these many times with bootstrap samples, we can get a good estimate of overfitting when we trained on the entire data set.

  4. If we are happy with how little overfitting there is, then we should use the model that was trained on the entire data set, which we now consider validated.

At no point do we adjust the coefficients.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.