2
$\begingroup$
  • In this excellent reply, Demetri Pananos estimates a regularized logistic regression model via cross-validation, then bootstraps the sample to derive an optimism-corrected calibration curve for that model. Each iteration uses the same model. The only thing that changes is the sample.

  • In another great reply to a similar question, Pananos develops one bootstrapped routine within which he estimates a model to get optimism-corrected performance metrics. In each iteration, both the estimating model and the sample change.

I'm not sure if these two approaches would yield similar corrections for the same dataset. On one hand, bootstrapping emphasizes that it's the strategy, and not the model, that should be repeatedly estimated. Changes in the model specification is a feature here, not a bug. On the other hand, if the bias correction is for a specific model, then why should we allow the model to change between iterations?

Is one approach necessarily better than the other?

$\endgroup$

1 Answer 1

1
$\begingroup$

Each iteration uses the same model. The only thing that changes is the sample.

That's not true, although it might be unclear from some of the code I present. Take a look at the second to last code block from that answer.

enter image description here

You'll see that I am re-performing grid search cross validation on the bootstraps. Hence, the sample and the model can change.

$\endgroup$
5
  • $\begingroup$ Thanks! I missed it when reading through your response. Does this mean that the original model is first validated over the entire data in a 10-fold CV 100 times, then re-estimated and revalidated inside the bootstrap nsim (500) times? $\endgroup$
    – wahid
    Commented Aug 24, 2022 at 0:45
  • $\begingroup$ Do we lose anything by performing the hyperparameter validation separately from the optimism correction? $\endgroup$
    – wahid
    Commented Aug 24, 2022 at 0:51
  • $\begingroup$ @wahid. The model is selected using the repeated cross validation. That procedure is repeated for every bootstrap because we want to validate the process of selecting the model, not any one model in particular. The hyperparameter selection is part of the model, so it is performed during the estimate of the optimism. $\endgroup$ Commented Aug 24, 2022 at 1:44
  • $\begingroup$ If I understand you correctly, does that mean lines 44:46 of your gist are for demonstration purposes, only? $\endgroup$
    – wahid
    Commented Aug 24, 2022 at 1:59
  • 1
    $\begingroup$ @wahid Yes, I believe so. $\endgroup$ Commented Aug 24, 2022 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.