I'm looking to create an ensemble of CNNs for segmentation. Do you know how many CNNs I should have in my ensemble? Would I be tuning this or is there a general amount that I should have? I noticed that papers tend to have between 10-30 CNNs. How about 100? Or 1000? Are the limitations here just resource constraints? Just from reading online it seems there's a drop-off after a certain point, which makes sense because if you assume using a single classifier may vary from the mean from a certain amount, increasing the number of samples would more closely approximate the mean up until you reach a good sample size, if you're doing model averaging. Is there any resource for the number of classifiers used for ensemble methods, especially for segmentation? Also, clearly there are other types of ensemble methods as well - is there any systematic study on using different types of ensemble methods with CNNs (bagging, adaboost, etc.) and which are advantageous, especially for segmentation?
-
$\begingroup$ Here are some slides in regards to the effect of the number of classifiers on test error in bagging and boosting: ic.unicamp.br/~wainer/cursos/1s2012/mc906/ensembles.pdf. Based on the slides, it seems to suggest there's a dropping off point after a certain sample size so perhaps I can evaluate my ensemble in 5-10 classifier increments or something. I guess it wouldn't hurt to potentially have more classifiers obviously than less. $\endgroup$– bobCommented Sep 9, 2017 at 1:04
1 Answer
I am unfamiliar with using ensembles of CNNS, but you might be interested in Hinton, Geoffrey E., et al. "Improving neural networks by preventing co-adaptation of feature detectors." arXiv preprint arXiv:1207.0580 (2012).
Dropout can be seen as an extreme form of bagging in which each model is trained on a single case and each parameter of the model is very strongly regularized by sharing it with the corresponding parameter in all the other models.
So, if you equate dropout to bagging, then ensemble methods are used extensively for neural networks in the form of dropout. It is probably much easier (and less expensive) to tune the degree of dropout than to tune an actual ensemble of CNNs.
If you do want to use an ensemble, the number of estimators would have to be chosen as a hyperparameter using a validation set or cross validation because the best number of estimators would be highly dependent on the data (although perhaps if there are networks trained on images you could use their choice as it seems that insights from working with one set of images tend to generalize, perhaps because processing pixels is kind of more general than processing structured attributes). Finding the number of estimators would be difficult because now the hyperparameters of each estimator would be conditioned on the number of estimators. This might not be so bad when they are decision trees, but when they are CNNs it might be really difficult to train.
In terms of boosting, generally that is used to combine weak estimators while CNNs are high capacity models (although it might be a good idea to boost them, don't know).
