# How to compare models using bootstrap optimism adjusted prediction error?

Let's say I'm comparing the prediction error of two different models.

For illustration purposes we'll use a toy example. I've generated 5 bootstrap samples and fit Model A and Model B to each bootstrap, then calculated the difference between the prediction error and apparent error (the "optimism") for Models A and B on each bootstrap sample.

Model A has an average residual squared error of 10.0 and Model B has an average residual squared error of 9.2.

\begin{array} {} &\text{Bootstrap Sample:} &\text{Model A Optimism:} &\text{Model B Optimism:}\\\hline &\text{1}&\text{1.1}&\text{2.1}\\ &\text{2}&\text{1.2}&\text{2.2}\\ &\text{3}&\text{1.3}&\text{2.3}\\ &\text{4}&\text{1.4}&\text{2.4}\\ &\text{5}&\text{1.5}&\text{2.5}\\ &\text{Average}&\text{1.3}&\text{2.3}\\ \end{array}

In order to compare the adjusted prediction error estimates for Models A and B, do we calculate (average residual squared error) - (average optimism) for each model: 10.0 + 1.3 = 11.3 for Model A and 9.2 + 2.3 = 11.5 for Model B, concluding that Model A is the better performer?

Or do we calculate (average residual squared error) - (optimism for bootstrap sample) for each bootstrap sample and then compare the range of adjusted prediction error estimates? In this case Model A's adjusted prediction errors would range from 11.1 to 11.5 and Model B's adjusted prediction errors would range from 11.3 to 11.7, a significant overlap.

## 1 Answer

I can't comment ('low' reputation) so I'm posting this as an answer.

Take a look at this page and at the paper mentioned therein. You should also check the books linked there; both are standard references in machine/statistical learning. I'm going through this problem myself and these references have been quite useful.

• Readers should be aware of the massive potential bias with optimism corrected bootstrapping with higher numbers of features. Reproducible code and simulations are here, it is also quite easy to show where the problem in the method originates from because there is an information leak between training and test datasets: intobioinformatics.wordpress.com/2018/12/29/…. – Christopher John Feb 26 '19 at 11:16