I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any examples of dropout being practically implemented into a stacked autoencoder.

Where would the dropout layer(s) go, between every layer, only after the input layer, is anyone able to let me know/provide some resource implementing this?

  • $\begingroup$ Many of the uses of stacked auto encoders I've seen add noise to the input data. If that's your use case, have you tried increasing the noise instead? That should also decrease the networks ability to overfit. $\endgroup$ – Wayne Feb 6 at 14:44
  • $\begingroup$ Yes, I have seen this and I can easily implement this, but I was also curious about using dropout as its a very similar concept. $\endgroup$ – TwoRice Feb 6 at 15:05
  • $\begingroup$ It really does depend on use case. Noise is de riguer for some applications and probably not a good idea for others. At a minimum, adding noise to the input occurs early in the process (of that leg of the data), while dropout can in general occur at almost any level of a network. $\endgroup$ – Wayne Feb 6 at 15:16

We have tried adding it in few different ways:

  1. Add only after input layer. That will make some inputs zero
  2. Add after input and every encoder layer. That will make some inputs and encoded outputs zero. We didn't want decoder layers to lose information while trying to deconstructing the input.

However, we could not eliminate the overfitting completely. We have also tried adding noise to input data (Denoising AE), adding regularization (Sparse AE) to encoding layers. But still our hand made features performed better than AE created features.

  • $\begingroup$ Welcome to our site. Could you explain how your post answers the question in this thread? $\endgroup$ – whuber Feb 16 at 19:51
  • 1
    $\begingroup$ It explains where to add dropout layer in a model. Also some additional options instead of dropout. $\endgroup$ – Jemshit Iskenderov Feb 16 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.