Timeline for Dropout makes performance worse
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 6, 2020 at 5:02 | vote | accept | Yuri | ||
| Aug 23, 2017 at 3:34 | comment | added | Yuri | @shimao "When the network is small relative to the dataset" - the IRIS dataset has 150 examples, but my model has much more parameters than that. | |
| Aug 23, 2017 at 3:31 | comment | added | Yuri | So in all those Keras examples Dropout is added just because it can be added? Also you are saying that if I increase my model's capacity (make it comparable to the size of the dataset) then by adding dropout I can get a better result than for the original (small) model? | |
| Aug 23, 2017 at 3:08 | comment | added | Matthew Drury | I think it's mostly 2 in this case. For smallish or medium sized networks on the size a hobbyist may train using a basic machine, ridge penalization seems to almost always be sufficient. It was only when people started training really deep networks that dropout was discovered. | |
| Aug 23, 2017 at 2:45 | comment | added | Yuri | Thank you for your answer! As I mentioned in my question most state of the art results use dropout, for example, check out this paper arxiv.org/abs/1707.05589 Also dropout is more that just a regularization technique, the idea behind Dropout is to train an ensemble of DNNs and average the results of the whole ensemble instead of train a single DNN. 1. In all the examples people put dropout before the last layer, this paper is an example arxiv.org/abs/1707.05589 2. I tried bigger networks - the same story. 3. I tried to train longer - the same story. | |
| Aug 23, 2017 at 1:58 | history | answered | shimao | CC BY-SA 3.0 |