Dropout makes performance worse I am playing with dropout since all state of the art results in machine learning seem to be using it (for example, see here). I am familiar with all the guidelines (train longer, increase capacity of the model, use higher learning rates), but still cannot see it working. I've tried several different examples: CNN for IMDB, CNN for MNIST, MLP for MNIST, MLP for IRIS, and turning off dropout makes all my results better even though the default configurations have dropout (taken from the Keras examples). For example, I am attaching my results for one of the models trained on the IRIS dataset. The configuration without dropout has clearly the best performance. 
What am I missing?
The code for the IRIS example is here.
 A: Dropout is a regularization technique, and is most effective at preventing overfitting. However, there are several places when dropout can hurt performance.


*

*Right before the last layer. This is generally a bad place to apply dropout, because the network has no ability to "correct" errors induced by dropout before the classification happens. If I read correctly, you might have put dropout right before the softmax in the iris MLP.

*When the network is small relative to the dataset, regularization is usually unnecessary. If the model capacity is already low, lowering it further by adding regularization will hurt performance. I noticed most of your networks were relatively small and shallow.

*When training time is limited. It's unclear if this is the case here, but if you don't train until convergence, dropout may give worse results. Usually dropout hurts performance at the start of training, but results in the final ''converged'' error being lower. Therefore, if you don't plan to train until convergence, you may not want to use dropout.
Finally, I want to mention that as far as I know, dropout is rarely used nowaways, having been supplanted by a technique known as batch normalization. Of course, that's not to say dropout isn't a valid and effective tool to try out.
