2
$\begingroup$

I have a deep learning classification pipeline in which I had created 10 independent train/val/test splits. The pipeline uses large images which must be broken up into "tiles" which are assigned the label of the original image. I have a script which gives me the AUC and 95% boostrapped confidence interval on a given test set. After running on all 10 splits, I have 10 AUCs and 10 CIs. I was wondering if there was a proper way to aggregate these and generate 1 single statistic AUC and CI.

Intuitively I would think to get the mean AUC, the mean lower bound CI and mean upper bound CI, but I am not sure if that is correct and what to call that if it is correct.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Assuming that the test sets are independent and come from same distribution, the solution is pretty simple: combine the bootstrap samples from all the trials and calculate the interval using the combined samples (i.e. quantiles of the combined samples). It simply sounds like all your bootstrap samples come from the same distribution, so there’s no reason why you couldn’t combine them and consider as a single sample.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think the test sets are independent. Just to be clear, the notion of independence here doesn't care if there are a sample appears in multiple test sets as long as the way one test set was generated is independent from the others, correct? $\endgroup$
    – Mattreex
    Commented May 26, 2021 at 18:33
  • $\begingroup$ @Mattreex when you use bootstrap or something like cross-validation in general this could happen, doesn’t it? Unless I’m misunderstanding the procedure you used? $\endgroup$
    – Tim
    Commented May 26, 2021 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.