0
$\begingroup$

I have worked on project where we evaluated transfer learning of several CNNs for medical dataset. We have used 5-fold cross validation and now are about to report results. I have taken the relevant metrics (F1 and Kappa) and averaged them across the folds for each model getting average numbers.

I have seen quite a few papers with confidence interval estimation in classic ML, but not so much in the area of Deep Learning. If I understood correctly, calculating CI require a lot of repetitions and could be unsuitable for Deep Learning. I am personally using dataset with over 10k images where one fold of 400 epochs takes approx. 16 hours. I think the 5 folds I created aren't significant and can't be used for CI creation, right?

If so, what should one report with F1 and Kappa to give better idea about the results? Std?

Thanks

Improve this question
$\endgroup$
3
  • $\begingroup$ You can still do it but your confidence intervals might be very wide (and maybe wrong if your distributional assumption is wrong) $\endgroup$
    – Akababa
    Jul 27, 2019 at 21:34
  • $\begingroup$ @Akababa Thank you. I have around 10k of examples in 72 classes and they are nowhere near normally distributed. Is there any test you may recommend for such case? $\endgroup$
    – sob3kx
    Jul 28, 2019 at 8:00
  • $\begingroup$ I don't have time now to write an anwer, but my answer at stats.stackexchange.com/a/404841 may help already. Your situation sounds interesting to me (in the sense that I could imagine a fruitful collaboration towards constructing these confidence intervals - but I'm drowning in work right now, so that such a collaboration would be too slow for your current publication) $\endgroup$
    – cbeleites unhappy with SX
    Jul 28, 2019 at 9:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.