6
$\begingroup$

In the last course of the Deep Learning Specialization on Coursera from Andrew Ng, you can see that he uses the following sequence of layers on the output of an LSTM layer:

Dropout -> BatchNorm -> Dropout.

To be honest, I do not see any sense in this. I don't think dropout should be used before batch normalization, depending on the implementation in Keras, which I am not completely familiar with, dropout either has no effect or has a bad effect.

I might be missing something here, though, and if anyone has any knowledge of why something like this could be useful, I'd love to hear from them.

Improve this question
$\endgroup$
6
  • $\begingroup$ Are you asking why you should use Dropout in that configuration or why you would ever use a Dropout layer? $\endgroup$
    – blackplant
    Commented Feb 8, 2018 at 12:28
  • $\begingroup$ I am asking what is the benefit of using dropout before batch normalization. $\endgroup$
    – Aleksandar Jovanovic
    Commented Feb 8, 2018 at 13:29
  • 2
    $\begingroup$ Did you try to discuss this in the course (lively) forum, where teaching assistants and mentors are always available and happy to assist?? $\endgroup$
    – desertnaut
    Commented Feb 8, 2018 at 16:37
  • $\begingroup$ While the combination DO->BN->DO looks strange, I cannot follow why DO followed by BN strikes you as a bad idea. Can you elaborate? $\endgroup$
    – pietz
    Commented Feb 8, 2018 at 23:52
  • $\begingroup$ @pietz Well, basically, if dropout removes some of the neuron activations in the previous layer, wouldn't that give an invalid estimate of the mean and variance for that activation? The normalization would thus be invalid. On the other hand, if it does not do such a thing, i.e. if it ignores dropout, than basically dropout has no purpose in here and should be removed. $\endgroup$
    – Aleksandar Jovanovic
    Commented Feb 9, 2018 at 9:48

2 Answers 2

Reset to default
2
$\begingroup$

The way I see it, it introduces much more noise into the model that a single batch normalization layer. But as shown in https://arxiv.org/pdf/1801.05134.pdf, dropout doesn't go well with batch normalization. Noone says Andrew Ng is infallible.

Improve this answer
$\endgroup$
2
$\begingroup$

Your intuition that dropout changes the statistics is discussed in detail in this arxiv paper (to my knowledge, neither peer-reviewed, nor properly published anywhere else.) I do not vouch for any of their proposed solutions.

A similar paper that was submitted to ICLR 1017, narrowly rejected on the grounds of too incremental an improvement. Likewise, I do not vouch for proposed solutions.

For what it's worth, my intuition is the same. I tentatively advance the hypothesis that this is in the sour spot of, "mostly understood by the research community and so not often mentioned, analyzed, or written up; but also not quite obvious, especially to beginners or those with a kitchen sink mentality." It surprising that Ng, who is not a beginner, would fall prey to that, but no one is perfect.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.