Evaluate the performance of a model with bootstrap

Question

This question is about the application of the bootstrap rule The population is to the sample as the sample is to the bootstrap samples.
I have a small dataset about lung cancer.There are 160 patients with 60 events and the predictors are 700 radiomic features.
The goal of the analysis is to get a model which could accurately group new patients to a high-risk group or low-risk group. Since this is a small dataset, using bootstrap would be better than data splitting.
I use bootstrap to wrap all processes including the variable selection(Lasso), model fitting, and using some criterion to evaluate the model. During the bootstrap process(for example, 1000 replicates), resampling dataset have the same size as the original dataset. As the .632 rule suggests, 63.2% samples would be used as training data, and the out of bag samples would be used as test data for the evaluation step. For each round of bootstrap, the evaluating criterion I would like to collect is chi-square statistic produced by log-rank test between the predicted high-risk group and the predicted low-risk group. In my opinion, the evaluation score from the bootstrapping processes is doubly biased as it comes from resamples. And how could I infer the correct evaluation score of the population?

Answer 1

For your situation, your choices to use bootstrapping to evaluate your modeling and to wrap all your modeling processes within the bootstrapping are wise. There are just a few tweaks to your approach that should accomplish your goals.

As you note in a comment to a different answer, the .632 bootstrap* does not properly mimic the selection of your original sample from the population, as you "don't have out-of-bag samples for the original samples." With many choices of measures of model quality, however, you don't need to use the .632 bootstrap. Frank Harrell notes in this answer that the ordinary optimism bootstrap** works well except when you have a "discontinuous improper scoring rule" like the proportion correctly classified.

The ordinary optimism bootstrap nicely applies the bootstrap principle that you cite. The mean bias/optimism over multiple bootstrapped samples and corresponding models, each compared against the fits of those models on the entire original sample, represents the relationship of the bootstrapped samples to the original sample. That is also, by that bootstrapping principle, the expected value of the relationship of the original sample to the population. So you take whatever bias/optimism correction that you determine from the models on the bootstrapped samples versus the original sample, and apply it to your model based on the original sample to get the estimate for the population.

That said, a few warnings:

First, remember that this bootstrapping does not validate the model itself; rather it validates the process you used for model building. Those aren't necessarily the same thing.

Second, you should consider using a different criterion than the "chi-square statistic produced by log-rank test between the predicted high-risk group and the predicted low-risk group" for evaluating your original and bootstrapped models. Choosing "high-risk" and "low-risk" groups within each (bootstrapped) sample means that you are making a premature decision about classification when you have continuous information available about probabilities (logistic regression) or hazards (survival analysis). You will be much better served by evaluating a continuous probability or hazard model instead. For Cox models, for example, you can follow the glmnet package and optimize the partial likelihood deviance. To document the quality of the model, report the concordance, both original and optimism-adjusted, and the average slope optimism from your bootstrapping validation.

If you evaluate models based on a proper continuous criterion you only have to go through the bootstrapping validation once. If you evaluate based on some arbitrary choice of "high-risk" and "low-risk" cutoffs, then you would have to re-validate whenever you change your cutoff criterion. There is nothing to be gained by jumping straight ahead to the classification problem. If for some reason you are forced to dichotomize into high- and low-risk groups you would use the continuous probabilities/hazards to make informed choices that incorporate tradeoffs between different types of misclassification and information about other clinical variables.

Finally, think carefully about your use of LASSO for this problem. With 60 events, LASSO is presumably selecting (e.g., with choice of penalty based on optimizing partial likelihood deviance) only about 6 or so from among your 700 features. My suspicion is that the particular 6 or so features chosen vary dramatically among bootstrap samples. That might work for prediction, but it could leave you vulnerable to a particular feature that was extraordinarily related to outcome in your particular data sample versus its relation in the population as a whole. Consider other ways to pre-group predictors, without reference to outcome: for example, based on subject-matter knowledge, similarity clustering, or methods related to principal-components analysis. See chapter 4 of Harrell's Regression Modeling Strategies for such approaches. Ridge regression, which is principal-components regression with graded rather than all-or-none inclusion of principal components into the model (see ESL II, page 79) might also be useful.

---

*Evaluate models from bootstrap samples only on corresponding held-out cases.

**For each bootstrap sample, develop a model, apply it also to the entire original sample, determine the bias/optimism for that model between that bootstrapped sample and the entire original sample, then average the bias/optimism over many bootstrap samples.

Answer 2

Why not compute an accuracy value (# correct/total) for each out-of-bag sample instead of the other statistic you're computing?

Using the bootstrap method would then allow you to get a confidence interval and estimate of the model accuracy for the population.