Questions tagged [optimism]
The optimism tag has no usage guidance.
8
questions
0
votes
0
answers
22
views
Negative optimism resulting in better adjusted model performance
After carrying out 200 bootstraps to estimate the optimism around the net benefit of a model developed using stepwise BIC logistic regression, I'm getting a negative value meaning that when subtracted ...
- 41
0
votes
0
answers
16
views
What maximum value of AUC optimism could still be allowed to confirm that logistic regression model does not overfit?
I am not sure how to define that a statistical model does not overfit based on a difference between bootstrapped AUC and AUC calculated on all training data. In the literature I saw 2 approches. The ...
- 101
3
votes
1
answer
224
views
Estimating regression optimism using the bootstrap
I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am ...
- 1,035
0
votes
1
answer
207
views
Expected Optimism 0-1 Loss with 0-1 Response
Want to show that
$$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(g(x_i), Y_i)$$
For 0-1 loss function with 0-1 response.
Want I've done
$$op = l_{in}(g) - l(g)=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', g(...
- 101
0
votes
2
answers
633
views
Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?
My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking:
Please provide ...
- 251
2
votes
0
answers
262
views
What is a reliable way to obtain an optimism-correct AUC with confidence limits?
I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC).
I am trying to look at a method with ...
- 813
2
votes
0
answers
202
views
Optimism bootstrap with non-linear models
I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
- 31
2
votes
1
answer
889
views
In which scenarios are the in-sample error and training error NOT the same?
In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as:
$$
op\equiv Err_{in}-\overline{err}
$$
With the training error $\...
- 12.1k