3
$\begingroup$

I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm.

Here are my thoughts (I may be completely wrong):

Dropout:

From my understanding, this is used in supervised networks to curb overfitting by making sure each neuron represents something meaningful, rather than memorizing the training data.

For GANs, my guess is that dropout can be used to prevent the discriminator or generator from being too strong, therefore helping reduce the chance of mode collapse?

For VAE, I don't think dropout is useful? Not sure.

Batch norm:

From my understanding, batch norm reduces covariate shift inside of a neural network, which can be observed when you have different training and testing distributions.

Therefore, I think this isn't really applicable in GANs, since at test time we simply sample from a pre-set distribution (commonly used is $\mathcal{N}(0,1)$), the input data is usually from the same distribution.

On the other hand VAE can be affected by covariate shift, since the inputs at test time may have a different distribution than the training set.

$\endgroup$
3
  • 3
    $\begingroup$ I don't agree that this question should be closed. It's a perfectly fine statistical question that does not require any code to answer $\endgroup$
    – Firebug
    May 1, 2022 at 13:16
  • $\begingroup$ BatchNorm and, much more so, Dropout are not that commonplace as they were a few years ago. I'm not sure what is the current literature view on this, but for VAEs not using either was (still is?) the norm. $\endgroup$
    – Firebug
    May 1, 2022 at 13:18
  • $\begingroup$ @Firebug I have not caught up with current literature in ML/DL yet. What happened that caused both of these to drop out of being commonly used? Did something better come along? $\endgroup$ May 2, 2022 at 6:03

2 Answers 2

4
$\begingroup$

Dropout: I agree with comments saying that dropout has mostly been dropped (ha) in favor of other regularization techniques, especially as architectures have gone more fully convolutional (and dropout doesn't really work with conv layers). Also note that dropout and batch norm can have bad interactions with each other.

I don't think anyone really understands why batch norm helps - some have argued against "covariate shift" for example - How Does Batch Normalization Help Optimization?. So I don't this is a strike against using it in GANs.

Some GAN varieties like WGAN assume independence between samples in a batch, which is a good reason to avoid batch norm.

$\endgroup$
4
$\begingroup$

I would have preferred to just comment shimao's answer but my reputation does not suffice.

First, a small correction of the answer:

Some GAN varieties like WGAN assume independence between samples in a batch, which is a good reason to avoid batch norm.

As far as I know, this is not correct. The author's of WGAN actually encourage usage of batch norm [1,2]. However, the improved Wasserstein GAN (with Gradient Penalty), requires to omit or replace Batch Norm. They recommend Layer Norm as a replacement:

Our penalized training objective is no longer valid in this setting, since we penalize the norm of the critic’s gradient with respect to each input independently, and not the entire batch. To resolve this, we simply omit batch normalization in the critic in our models, finding that they perform well without it. Our method works with normalization schemes which don’t introduce correlations between examples. In particular, we recommend layer normalization [3] as a drop-in replacement for batch normalization.

Second, I would like to add that the usage of differentially private training is another reason against the usage of Batch Norm, as described in Google's DP-fy ML paper:

However, BatchNorm uses current batch’ mean and standard deviation information to rescale each instance in the batch during the forward pass. This creates dependency between instances from the batch and makes it hard to reason about per-example sensitivity for DPSGD.

  1. https://github.com/martinarjovsky/WassersteinGAN/issues/14#issuecomment-283092028
  2. https://github.com/martinarjovsky/WassersteinGAN/pull/6#issuecomment-277738812
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.