Upon using the dropout technique, my cost function is spiking arbitrarily. Is this normal? If not, how do I avoid it?

I'm using a salt-and-pepper mask to drop out neurons at a dropout rate of 5%. I've scaled the weights while training, by a factor of 1-dropout rate.

Cost vs Epochs with dropout of 5%

  • $\begingroup$ Are you decreasing the learning rate as you are training? $\endgroup$ – Indie AI Dec 28 '15 at 17:39
  • $\begingroup$ @IndieAI : Yes, I am performing a learning rate decay. I corrected this problem for a Neural Network by multiplying the weights with a scale, but it seems to persist for autoencoders while using denoising technique $\endgroup$ – arch3r Feb 8 '16 at 9:37

A 5% drop-out rate implies that each neuron is "on" (not dropped) with probability 0.95. For a small number of neurons, you can expect that all of them will be "on" with high probability. This is why your curve is mostly smooth with some spiking: the spikes correspond to the (somewhat rare) cases where one or more neurons is "off".

Moreover, because the dropout is so small, it's likely that the network is trained in such a way that it "expects" that all of the neurons are present -- so losing one or more neurons is reducing the network's effectiveness.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.