Optimism bootstrap with non-linear models - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-17T17:05:39Z https://stats.stackexchange.com/feeds/question/374906 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/374906 2 Optimism bootstrap with non-linear models thebigspin https://stats.stackexchange.com/users/225533 2018-11-01T23:30:39Z 2018-11-02T23:34:22Z <p>I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper scoring metric (log-likelihood/cross-entropy loss). </p> <p>A simplified hypothetical case appears to demonstrate this. Consider a binary case, with an overfit classifier. In this example, for an element in the training data, the classifier will predict the correct class with 0.99 probability. However, the classifier has fit purely to noise, which has no true correlation with the output class, such that when it is used to predict out-of-training-sample data, it predicts the correct class at 0.5 probability. This could occur for an arbitrary non-linear classifier. </p> <p>Using the notation from Frank Harrell's 1996 <a href="https://www.ncbi.nlm.nih.gov/pubmed/8668867" rel="nofollow noreferrer">paper</a>, the apparent per-sample cross-entropy loss (<span class="math-container">$D_{app}$</span>) of the classifier is <span class="math-container">$-\ln(0.99) = 0.004$</span>. However, for out of sample, unseen data the per-sample cross-entropy loss <span class="math-container">$D_{true}$</span> is <span class="math-container">$-\ln(0.5) = 0.3$</span>. As such, <span class="math-container">$$D_{boot} = D_{app} = 0.004,$$</span> and <span class="math-container">$$D_{orig} \approx 0.632*D_{app} + 0.368*D_{true} \approx 0.1.$$</span></p> <p>(Edit: Note that I have used 0.632 and 0.368 in the above equation to provide the analytical expectation of <span class="math-container">$D_{orig}$</span>, but this would normally be directly measured by fitting the model to the bootstrap sample, and using it to predict the original data. I am using the standard optimism bootstrap, and not the .632 or .632+ bootstrap methods).</p> <p>This gives an optimism <span class="math-container">$O$</span> of <span class="math-container">$$O = D_{boot}-D_{orig} \approx -0.096,$$</span></p> <p>which in turn gives an optimism corrected performance <span class="math-container">$D_{est}$</span> of <span class="math-container">$$D_{est} = D_{app} - O = 0.1.$$</span> This is clearly much lower than the true expected <span class="math-container">$D_{true}$</span> of <span class="math-container">$0.3$</span>.</p> <p>Obviously this is an extreme and overly simplified example, but it demonstrates the result I have observed in my results.</p> <p>Have I misunderstood the application of the optimism bootstrap, or is there another explanation? (With specific reference to @FrankHarrel's answers <a href="https://stats.stackexchange.com/a/96833/225533">here</a> and <a href="https://stats.stackexchange.com/a/71189/225533">here</a>)</p>