Statistically prove minimum number of classifiers to classify 12 datasets I'm currently developing neural networks to classify medical images. One dataset contains 2 classes of images (Images which can be used and cannot used). So my model will check whether given image can be used or not.This is a binary classification.
Likewise, I have 12 datasets (Each has 2 class images) to classify. All these 12 datasets are similar but not identical. So I thought to make 12 neural networks to perform classifications.
But one model might be 2GB in size and having 12 models will arise a storage problem. So I need to find out minimum number of models should I train to evaluate all these 12 datasets.
What I did
I already trained one neural network for one dataset and use that model to evaluate rest (11 datasets). So, I got 95% of prediction accuracy for 5 datasets and 6 datasets with lower than 90% accuracy.
Since I'm dealing with medical images, accuracy of classification should be high like 98%. So I'd like to know the statistical approach to find out the best possible minimum number of models for this? Can I use methods like ROC ? 
 A: How about you attack this problem in a hierarchical fashion. Say you train one network on the images. It gets around 95%, but you want atleast 98%. Then, what I would try to do, is to take that 5% that was misclassified, and then create a network that is trained on that data alone. Hopefully, this will further cut down on the number of misclassifications (although probably not as good at 95% accuracy). Now, we need a way to tell if the image belongs in the former class or the latter class. There are a few ways to do this, but as a starting point, you could train a third nerual net to find the best allocation of images to maximize accuracy. Note I don't mean to train all 3 simultaneously, as that would probably be very expensive, but to train them sequentially. Or if you can find common features between the classes, you could build something like a bayesian net which determines the best allocation.
A: There isn't really anything to prove as having "best possible minimum number of division" is a matter taste. It mostly depending on the assumptions you are willing to make.
There is a tradeoff between the amount of models and individual model accuracy. This is due to the approximation-estimation error tradeoff discussed later. 
Practically:


*

*Decide ahead of time on possible divisions of the datasets. 

*Choose between the divisions by cross validation.

*A model for a division will be composed of submodels, one for each group in the division. Each submodel is trained using all training data in all the datasets of the group.


Approximation-Estimation Error Tradeoff
In general, since you are mostly interested in a specific model, I think the right way to consider your performance is by the standard decomposition of the generalization error into approximation and estimation:
$$
Err_{gen} = Err_{app} + Err_{est}
$$
Where $Err_{app}$, or approximation error, is the lower error possible in the model, as no one knows if your model is able to actually represent the data. The estimation error second error is how close can you get to the best model.
You can lower $Err_{app}$ by choosing a better model. Your intuition did take this error into account, as you tried to make the model able to capture all the data. 
However, dividing the data into smaller dataset, each for a different algorithm, will cause each algorithm to be trained by less data. Usually, the probability to achieve a bad classifier, with estimation error bigger than a fixed value, shrinks exponentially with the amount of data.
So we have a tradeoff here - the more models, the less approximation error, but bigger estimation error (as we train on less data). While we could calculate the VC dimension of the different models, we can't know ${Err}_{app}$, unless we have infinite data. Thus, it seems we can't calculate which division minimizes ${Err}_{gen}$.
However, we can empirically decide which amount seems to gives the best classifier by cross validation. The possible divisions should be thought of ahead of time, as adaptive data analysis is dangerous. Finally, note that if you plan to compare many divisions, we should ensure we have enough data - either by using bootstrap or by gather more data (better). 
