3
$\begingroup$

I'm creating ensemble of neural networks for a simple binary classification task. Every neural network is generated and trained a bit differently (number of hidden layers, number of neurons per layer, learning rate etc.).

My question is, should I train these weaker networks and optimize each of them as much as possible or should I just use them as soon they reach "good enough" classification?

My "good enough" classification is around 80% since I can get it pretty fast without too much hassle of parameter optimization, but I can also reach 92-93% with some parameter tuning and it takes me 15-30 minutes to do it. What did you discover works better in general?

$\endgroup$
5
  • $\begingroup$ What kind of ensemble is it? $\endgroup$
    – Firebug
    Commented Jul 25, 2016 at 16:30
  • $\begingroup$ You might want to consider using dropout on a single network. In many ways, this acts as an ensemble of networks (as mentioned by Geoffrey Hinton). $\endgroup$ Commented Jul 25, 2016 at 17:51
  • $\begingroup$ @Firebug - what do you mean by "what kind of ensemble" ? At the moment I'm just taking majority vote and classifying it that way. John Yetter - I found dropout really not working in my case since my input is really sparse and I just couldn't tune dropout rate good enough. $\endgroup$
    – Drag0
    Commented Jul 25, 2016 at 19:45
  • 2
    $\begingroup$ There are many types of ensembles (e.g. bagging, stacking, boosting, etc.). That's what I meant. $\endgroup$
    – Firebug
    Commented Jul 25, 2016 at 19:49
  • $\begingroup$ @Firebug - I'm trying to create boosting ensemble. $\endgroup$
    – Drag0
    Commented Jul 25, 2016 at 19:52

1 Answer 1

3
$\begingroup$

For majority vote based approaches: with majority votes, there would not be any gain if all models would be completely equal. With ANNs you anyway usually have some random component (e.g. random initialization of connection weights), so each model should converge differently - which consequently is good for ensembles. Further, you could expand your majority vote approach towards a bagging approach and take a subset of samples/features for each individual model. This too makes models different, which might increase the overall prediction. So, your individual models could overfit to a certain extent (with the ensemble still being able to coop with it) if they overfit differently - caused by the models itself being different, as just discussed. If all goes well, the overfitting cancels itself out to a certain extent, which is the general variance reducing probability of most ensembles out there - but keep in mind that this usually requires quite some models to majority vote from. Therefore: you should likely tune each model even into slight overfitting - depending on how different they can become. In contrast, if you would end up with very similar models that all strongly overfit (e.g. fixed seeds for your ANNs that perfectly memorize your dataset), a majority vote won't help you either.

For boosting: in short - similar thoughts to the above. The idea is to combine multiple weak and unstable models into one strong and stable one. Weak model thereby means that it is not necessarily the most important thing that each individual model has good prediction performance on the majority of your samples, but should have a good prediction performance on a specific part of them. As boosting changes the weights assigned to samples over different iterations, each individual model will focus on different parts of data, namely those that are "currently difficult to predict". If you think about this for a moment, there again would not be any benefit if all models would be very similar to each other. Again, having different models that even overfit to certain parts of data slightly (caused by e.g. bagging-like mechanisms, built into many boosting approaches anyway) are beneficial. Be aware that because of assigning and modifying sample weights, boosting has a modified training procedure anyway, and you won't be able to tune as much as you would be with other ensemble approaches. Bottom line: yes, tuning again is OK - just don't produce too similar models (which will be hard with boosting anyway due to subsetting samples and changing sample weights).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.