I am a medical doctor who is working on a grant application. The project involves training deep learning models to classify medical images based on various different outcomes, for example diagnosis (where there may be about 8 classes), survival (binary categorisation, alive vs not) and some more complex "disease progression" outcomes. In the grant application (which is a generic application form) is indicates that details on the following should be provided:
- How the sample size was calculated, showing power calculations and including justification of effect size
- Circumstances in which power calculations are not appropriate to determine sample size
A correspondence with the grant committee who reviewed a one page summary of the project also mentioned that justification for training and validation sample sizes should be given.
My understanding, having trained a lot of different architectures (with some published work) is that dataset size is really an area of active research and usually its determination is by experimentation rather than objective calculation.
I know that statistical reviewers are on the review panel and I also know from experience that saying things like - "this is the way its usually done" just doesn't cut it. I am not a statistician.
I understand that its not a one size fits all and that the quantity of data required for model development is dependent on the complexity of the data, the complexity of the algorithm architecture and how well defined the classes in the classification problem are. However, is there a way to demonstrate with some mathematical rigor that you have at least attempted to evaluate the likely dataset size needed for training?