4
$\begingroup$

I'm searching for the best parameters of a classifier and I chose as comparison criterion the value of the specificity at 90% sensitivity (there are various reasons behind this choice). To assess the variability of the estimate I normally used a 10-fold Cross Validation, but having a highly unbalanced dataset, I feel that I was over-estimating the variance of my measures so I'm deciding to switch to Bootstrap (.632+ method). To calculate Confidence Intervals of my 10-fold Cross Validation results I used the classical formula

CI = [mean(spec) +/- t * std(spec) / sqrt(N)]

where spec are the 10 specificity values @ 90% sensitivity, N=10 and t is the critical value of Student's t distribution for alpha=0.05 and d.o.f.=9. mean and std are the sample mean and standard deviation of the 10 estimates.

My questions are:

  • is it correct to use this formula to calculate Confidence Intervals?
  • can I use it also for Boostrap estimates? And if not, why?

My second question arises from the fact that a lot of sources I checked suggest another formula, the "Basic Bootstrap" at the following Wikipedia link: https://en.wikipedia.org/wiki/Bootstrapping_(statistics)#Methods_for_bootstrap_confidence_intervals

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I'm assuming the notation of mean(spec) and sd(spec) refers to the notation of empirical means and SDs of the bootstrapped spec distribution. Is that correct? If so, unfortunately, that is not the correct way to calculate confidence intervals for the bootstrap (or for any other resampling based statistics) when the actual distribution under the null is not normal. The main strength of the bootstrap is the belief that it is robust to these types of distributional assumptions. The correct methods to calculate bootstraps for the general quantities have been discussed elsewhere on the site. The boot package has options to generate student, BCa, and percentile bootstrap confidence intervals. All of these are superior to normal approximations.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, your assumptions were correct. Is the above formula instead always valid for k-fold Cross Validation? If I were sure that my statistic follows a normal distribution could I use it? I'm asking because with a high number of resamplings the denominator gets pretty big. Thank you for your help! $\endgroup$
    – Marco
    Commented Jan 26, 2018 at 18:50
  • $\begingroup$ @Marco you should use the approach which is correct even if it is computationally expensive. You never actually know if a statistic actually follows a normal distribution or not. $\endgroup$
    – AdamO
    Commented Jan 26, 2018 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.