5
$\begingroup$

I’m trying to calculate 95% confidence intervals for the sensitivity and specificity of a decision model that I’m building.

I’ve split my dataset into 90/10 train and test sets. I’ve used the 90% train set to perform hyperparameter turning, and then used the optimal decision model selected from within the 90% train dataset to evaluate the 10% holdout dataset, which is fully independent, and not used in the hyperparameter tuning process.

My problem is, what's the best approach to obtain 95% confidence intervals for the training dataset? Should I bootstrap multiple subsets of the test data against the optimal model identified using hyperparameter tuning, and use those for the calculation? In example uses of bootstrapping that I found, different subsets of the train and test dataset are used for bootsrtapping. However, I don’t want to do that because I want my testing to be across a truly holdout (aka validation) dataset.

Improve this question
$\endgroup$
1
  • $\begingroup$ What are your sample sizes like? Why did you choose 90/10 instead of 75/25? What kind of a model is it in general? What parameter or summary statistic or you computing the confidence interval for? $\endgroup$
    – EngrStudent
    Commented Apr 8, 2022 at 11:16

2 Answers 2

Reset to default
0
$\begingroup$

You train your model on the 90% train dataset and then you bootstrap on the validation set (10%) and classify. Then rank your results and CI will be given from 26th and 975th value.

Improve this answer
$\endgroup$
2
  • $\begingroup$ See comment on other answer. $\endgroup$
    – toby544
    Commented May 27, 2023 at 9:37
  • 1
    $\begingroup$ How do you know that is a confidence interval? $\endgroup$
    – Frank Harrell
    Commented Mar 26 at 12:51
0
$\begingroup$

Generate different 10% validation 90% training splits. Run model generation on each training set. Determine model performance on accompanying testing set. Determine confidence intervals from resulting vector of metrics. For more indepth look see https://www.sisostds.org/DesktopModules/Bring2mind/DMX/API/Entries/Download?Command=Core_Download&EntryId=36208&PortalId=0&TabId=105 $^\dagger$.

$\dagger$ link dead

Improve this answer
$\endgroup$
1
  • $\begingroup$ Taking percentiles like that gives an interval of some kind, but I am not sure it is a confidence interval. Your linked paper does not mention cross validation, splitting, quantile, or percentile, which it should do if it is claiming that your idea is right. See this comment: stats.stackexchange.com/questions/100159/… $\endgroup$
    – toby544
    Commented May 27, 2023 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.