Is there a theoretical basis against using both L2 and Dropout regularization simultaneously for training a deep neural network? They are both related but could they be complementary if used together?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
The paper {1} that introduced dropout combined dropout with L2:
We found that dropout combined with max-norm regularization gives the lowest generalization error.
- {1} Srivastava, Nitish, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. "Dropout: a simple way to prevent neural networks from overfitting." Journal of Machine Learning Research 15, no. 1 (2014): 1929-1958. http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf
answered Oct 19, 2016 at 1:50
-
$\begingroup$ Thanks for the reference in the original paper! Sometimes it is best to go to the source. As a follow up, would there be an issue with using dropout + l2 + maxnorm simulteously? $\endgroup$– PylanderCommented Oct 19, 2016 at 21:34
-
$\begingroup$ The link doesn't work anymore $\endgroup$ Commented Feb 16, 2018 at 8:11
-
$\begingroup$ @MayurKulkarni Stack Exchange should take care of dead links, not individual users. $\endgroup$ Commented Feb 16, 2018 at 8:16