1
$\begingroup$

I am a medical doctor who is working on a grant application. The project involves training deep learning models to classify medical images based on various different outcomes, for example diagnosis (where there may be about 8 classes), survival (binary categorisation, alive vs not) and some more complex "disease progression" outcomes. In the grant application (which is a generic application form) is indicates that details on the following should be provided:

  • How the sample size was calculated, showing power calculations and including justification of effect size
  • Circumstances in which power calculations are not appropriate to determine sample size

A correspondence with the grant committee who reviewed a one page summary of the project also mentioned that justification for training and validation sample sizes should be given.

My understanding, having trained a lot of different architectures (with some published work) is that dataset size is really an area of active research and usually its determination is by experimentation rather than objective calculation.

I know that statistical reviewers are on the review panel and I also know from experience that saying things like - "this is the way its usually done" just doesn't cut it. I am not a statistician.

I understand that its not a one size fits all and that the quantity of data required for model development is dependent on the complexity of the data, the complexity of the algorithm architecture and how well defined the classes in the classification problem are. However, is there a way to demonstrate with some mathematical rigor that you have at least attempted to evaluate the likely dataset size needed for training?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Do you have a research hypothesis? E.g. machines are better than humans? If not, your research proposal is more exploratory than confirmatory. Standard statistical tools (google) are appropriate for the test set. The size of the training sample will be difficult to pick or argue a priori: obviously it's better to aim high than low. $\endgroup$ – Jim Mar 2 '18 at 10:49
  • $\begingroup$ For some outcomes there will be a clear comparator group - for example, a group of humans diagnose 100 patients. The algorithm is given the same task. Accuracy might evaluated against the majority vote of the humans on each case. Agreement might be assessed using Kappa. But for other outcomes, like predicting mortality - its a case of using standard survival stats on the algorithms prediction (dead or not). But is the amount of training data really just "get as much as you can"? $\endgroup$ – GhostRider Mar 2 '18 at 10:52
  • $\begingroup$ "The size of the training sample will be difficult to pick or argue a priori: obviously it's better to aim high than low" - this is the crux of my question.... $\endgroup$ – GhostRider Mar 2 '18 at 10:53
  • $\begingroup$ In Deep Learning, training requires a lot of data and takes takes a lot of time. You should have a look at the learning curve of your model to get a feeling for the size of a sufficiently large training set. $\endgroup$ – lnathan Mar 2 '18 at 11:08
  • $\begingroup$ I'm afraid there currently is very little known about this. One reason: the number of training examples needed differs widely per (supervised learning) task. If you do need such a number, look perhaps at papers that used a similar deep learning architecture for a similar learning task. $\endgroup$ – Jim Mar 2 '18 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.