Questions tagged [optimism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers

What maximum value of AUC optimism could still be allowed to confirm that logistic regression model does not overfit?

I am not sure how to define that a statistical model does not overfit based on a difference between bootstrapped AUC and AUC calculated on all training data. In the literature I saw 2 approches. The ...
zubenel's user avatar
  • 101
3 votes
1 answer

Estimating regression optimism using the bootstrap

I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am ...
Danny's user avatar
  • 1,035
0 votes
1 answer

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(g(x_i), Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in}(g) - l(g)=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', g(...
Casper Lindberg's user avatar
0 votes
2 answers

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...
Mohamed Rahouma's user avatar
2 votes
0 answers

What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with ...
LSC's user avatar
  • 773
2 votes
0 answers

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
thebigspin's user avatar
2 votes
1 answer

In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$ op\equiv Err_{in}-\overline{err} $$ With the training error $\...
Skander H.'s user avatar
  • 11.8k