Questions tagged [optimism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
48 views

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...
0
votes
0answers
31 views

What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with good performance and a plan to ...
0
votes
0answers
82 views

Model comparison when using Harrell's bootstrap-based optimism-correction Internal Validation procedure

I have 2 non-nested models that I would like to compare with respect to their predictive accuracy (concordance index). I have used Harrell's (Efron-Gong) bootstrap-based optimism-correction internal ...
2
votes
0answers
120 views

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
1
vote
1answer
146 views

In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$ op\equiv Err_{in}-\overline{err} $$ With the training error $\...