3
$\begingroup$

I have only heard about dropout being applied to training of neural networks. Could the same technique, in theory, be applied to any iterative ML algorithm? For example, in mini-batch training, each mini-batch iteration could randomly drop out some arbitrary features. Has this been tried for, say, logistic regression with SGD optimization? Any thoughts/opinions appreciated.


Wager et al. seem to follow the approach I suggested in the OP:

http://papers.nips.cc/paper/4882-dropout-training-as-adaptive-regularization.pdf

$\endgroup$
4
$\begingroup$

One simplest example of dropout being used in other algorithms is random forest: At each step of random forest, you randomly select only several of the features to split your trees, this is totally equivalent to randomly dropping out some of the features as in neural network.

$\endgroup$
  • 2
    $\begingroup$ This is an imperfect analogy though. While feature selection in random forests is random, you ultimately settle on features and splits that provide some minimal gain in accuracy. Whereas with dropout, you will keep all your neurons. $\endgroup$ – Alex R. Aug 30 '17 at 19:24
  • 1
    $\begingroup$ @AlexR. I partially agree. I think the similarity is stronger than you put here. If you have a large number of trees in random forest, it is highly probable that those not-so-useful features are still used in some of the trees at some splitting, so you still keeps all your "neurons" in random forest, but these not-so-useful features are not used as often as others. $\endgroup$ – DiveIntoML Aug 30 '17 at 19:57
  • 1
    $\begingroup$ +1. To your point though, what you describe with random forests is somewhat more related to DropConnect where you dropout weights in individual neurons. For ensembles of random forests, the analogy of dropout would be to random drop out entire trees and use the remaining ones to compute the output (as an average, for example) $\endgroup$ – Alex R. Aug 30 '17 at 20:41
  • $\begingroup$ @AlexR. Completely agree with you on this. $\endgroup$ – DiveIntoML Aug 31 '17 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.