4
$\begingroup$

Upon using the dropout technique, my cost function is spiking arbitrarily. Is this normal? If not, how do I avoid it?

I'm using a salt-and-pepper mask to drop out neurons at a dropout rate of 5%. I've scaled the weights while training, by a factor of 1-dropout rate.

Cost vs Epochs with dropout of 5%

Improve this question
$\endgroup$
2
  • $\begingroup$ Are you decreasing the learning rate as you are training? $\endgroup$
    – Indie AI
    Commented Dec 28, 2015 at 17:39
  • $\begingroup$ @IndieAI : Yes, I am performing a learning rate decay. I corrected this problem for a Neural Network by multiplying the weights with a scale, but it seems to persist for autoencoders while using denoising technique $\endgroup$
    – arch3r
    Commented Feb 8, 2016 at 9:37

1 Answer 1

Reset to default
1
$\begingroup$

A 5% drop-out rate implies that each neuron is "on" (not dropped) with probability 0.95. For a small number of neurons, you can expect that all of them will be "on" with high probability. This is why your curve is mostly smooth with some spiking: the spikes correspond to the (somewhat rare) cases where one or more neurons is "off".

Moreover, because the dropout is so small, it's likely that the network is trained in such a way that it "expects" that all of the neurons are present -- so losing one or more neurons is reducing the network's effectiveness.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.