Questions tagged [optimism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
59 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(g(x_i), Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in}(g) - l(g)=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', g(...
0
votes
2answers
79 views

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...
1
vote
0answers
77 views

What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with ...
2
votes
0answers
136 views

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
1
vote
1answer
246 views

In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$ op\equiv Err_{in}-\overline{err} $$ With the training error $\...