I have worked on project where we evaluated transfer learning of several CNNs for medical dataset. We have used 5-fold cross validation and now are about to report results. I have taken the relevant metrics (F1 and Kappa) and averaged them across the folds for each model getting average numbers.

I have seen quite a few papers with confidence interval estimation in classic ML, but not so much in the area of Deep Learning. If I understood correctly, calculating CI require a lot of repetitions and could be unsuitable for Deep Learning. I am personally using dataset with over 10k images where one fold of 400 epochs takes approx. 16 hours. I think the 5 folds I created aren't significant and can't be used for CI creation, right?

If so, what should one report with F1 and Kappa to give better idea about the results? Std?


  • $\begingroup$ You can still do it but your confidence intervals might be very wide (and maybe wrong if your distributional assumption is wrong) $\endgroup$
    – Akababa
    Jul 27, 2019 at 21:34
  • $\begingroup$ @Akababa Thank you. I have around 10k of examples in 72 classes and they are nowhere near normally distributed. Is there any test you may recommend for such case? $\endgroup$
    – sob3kx
    Jul 28, 2019 at 8:00
  • $\begingroup$ I don't have time now to write an anwer, but my answer at stats.stackexchange.com/a/404841 may help already. Your situation sounds interesting to me (in the sense that I could imagine a fruitful collaboration towards constructing these confidence intervals - but I'm drowning in work right now, so that such a collaboration would be too slow for your current publication) $\endgroup$ Jul 28, 2019 at 9:47


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.