26
$\begingroup$

I am playing with dropout since all state of the art results in machine learning seem to be using it (for example, see here). I am familiar with all the guidelines (train longer, increase capacity of the model, use higher learning rates), but still cannot see it working. I've tried several different examples: CNN for IMDB, CNN for MNIST, MLP for MNIST, MLP for IRIS, and turning off dropout makes all my results better even though the default configurations have dropout (taken from the Keras examples). For example, I am attaching my results for one of the models trained on the IRIS dataset. The configuration without dropout has clearly the best performance. Effect of dropout on MLP

What am I missing?

The code for the IRIS example is here.

Improve this question
$\endgroup$
8
  • $\begingroup$ Doesn't work for me either. Perhaps dependent on features of the problem? $\endgroup$
    – generic_user
    Commented Aug 23, 2017 at 0:16
  • $\begingroup$ @generic_user But the same is true for the MNIST dataset, where features are well known. $\endgroup$
    – Yuri
    Commented Aug 23, 2017 at 0:18
  • $\begingroup$ How deep were your networks? $\endgroup$
    – Matthew Drury
    Commented Aug 23, 2017 at 0:28
  • $\begingroup$ @MatthewDrury I considered several examples from the Keras repository, they have only few layers. $\endgroup$
    – Yuri
    Commented Aug 23, 2017 at 0:33
  • $\begingroup$ The main point of dropout is to prevent overfitting. So to see how well it is doing, make sure you are only comparing test data loss values, and also that without using dropout you are getting overfitting problems. Otherwise there may not be much reason to use it $\endgroup$
    – Frobot
    Commented Aug 29, 2017 at 4:15

1 Answer 1

Reset to default
61
$\begingroup$

Dropout is a regularization technique, and is most effective at preventing overfitting. However, there are several places when dropout can hurt performance.

  1. Right before the last layer. This is generally a bad place to apply dropout, because the network has no ability to "correct" errors induced by dropout before the classification happens. If I read correctly, you might have put dropout right before the softmax in the iris MLP.

  2. When the network is small relative to the dataset, regularization is usually unnecessary. If the model capacity is already low, lowering it further by adding regularization will hurt performance. I noticed most of your networks were relatively small and shallow.

  3. When training time is limited. It's unclear if this is the case here, but if you don't train until convergence, dropout may give worse results. Usually dropout hurts performance at the start of training, but results in the final ''converged'' error being lower. Therefore, if you don't plan to train until convergence, you may not want to use dropout.

Finally, I want to mention that as far as I know, dropout is rarely used nowaways, having been supplanted by a technique known as batch normalization. Of course, that's not to say dropout isn't a valid and effective tool to try out.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for your answer! As I mentioned in my question most state of the art results use dropout, for example, check out this paper arxiv.org/abs/1707.05589 Also dropout is more that just a regularization technique, the idea behind Dropout is to train an ensemble of DNNs and average the results of the whole ensemble instead of train a single DNN. 1. In all the examples people put dropout before the last layer, this paper is an example arxiv.org/abs/1707.05589 2. I tried bigger networks - the same story. 3. I tried to train longer - the same story. $\endgroup$
    – Yuri
    Commented Aug 23, 2017 at 2:45
  • $\begingroup$ I think it's mostly 2 in this case. For smallish or medium sized networks on the size a hobbyist may train using a basic machine, ridge penalization seems to almost always be sufficient. It was only when people started training really deep networks that dropout was discovered. $\endgroup$
    – Matthew Drury
    Commented Aug 23, 2017 at 3:08
  • 1
    $\begingroup$ So in all those Keras examples Dropout is added just because it can be added? Also you are saying that if I increase my model's capacity (make it comparable to the size of the dataset) then by adding dropout I can get a better result than for the original (small) model? $\endgroup$
    – Yuri
    Commented Aug 23, 2017 at 3:31
  • 2
    $\begingroup$ @shimao "When the network is small relative to the dataset" - the IRIS dataset has 150 examples, but my model has much more parameters than that. $\endgroup$
    – Yuri
    Commented Aug 23, 2017 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.