# How to compare models using bootstrap optimism adjusted prediction error?

Let's say I'm comparing the prediction error of two different models.

For illustration purposes we'll use a toy example. I've generated 5 bootstrap samples and fit Model A and Model B to each bootstrap, then calculated the difference between the prediction error and apparent error (the "optimism") for Models A and B on each bootstrap sample.

Model A has an average residual squared error of 10.0 and Model B has an average residual squared error of 9.2.

\begin{array} {} &\text{Bootstrap Sample:} &\text{Model A Optimism:} &\text{Model B Optimism:}\\\hline &\text{1}&\text{1.1}&\text{2.1}\\ &\text{2}&\text{1.2}&\text{2.2}\\ &\text{3}&\text{1.3}&\text{2.3}\\ &\text{4}&\text{1.4}&\text{2.4}\\ &\text{5}&\text{1.5}&\text{2.5}\\ &\text{Average}&\text{1.3}&\text{2.3}\\ \end{array}

In order to compare the adjusted prediction error estimates for Models A and B, do we calculate (average residual squared error) - (average optimism) for each model: 10.0 + 1.3 = 11.3 for Model A and 9.2 + 2.3 = 11.5 for Model B, concluding that Model A is the better performer?

Or do we calculate (average residual squared error) - (optimism for bootstrap sample) for each bootstrap sample and then compare the range of adjusted prediction error estimates? In this case Model A's adjusted prediction errors would range from 11.1 to 11.5 and Model B's adjusted prediction errors would range from 11.3 to 11.7, a significant overlap.