4
$\begingroup$

I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any examples of dropout being practically implemented into a stacked autoencoder.

Where would the dropout layer(s) go, between every layer, only after the input layer, is anyone able to let me know/provide some resource implementing this?

Improve this question
$\endgroup$
3
  • $\begingroup$ Many of the uses of stacked auto encoders I've seen add noise to the input data. If that's your use case, have you tried increasing the noise instead? That should also decrease the networks ability to overfit. $\endgroup$
    – Wayne
    Feb 6, 2019 at 14:44
  • $\begingroup$ Yes, I have seen this and I can easily implement this, but I was also curious about using dropout as its a very similar concept. $\endgroup$
    – TwoRice
    Feb 6, 2019 at 15:05
  • $\begingroup$ It really does depend on use case. Noise is de riguer for some applications and probably not a good idea for others. At a minimum, adding noise to the input occurs early in the process (of that leg of the data), while dropout can in general occur at almost any level of a network. $\endgroup$
    – Wayne
    Feb 6, 2019 at 15:16

1 Answer 1

Reset to default
2
$\begingroup$

We have tried adding it in few different ways:

  1. Add only after input layer. That will make some inputs zero
  2. Add after input and every encoder layer. That will make some inputs and encoded outputs zero. We didn't want decoder layers to lose information while trying to deconstructing the input.

However, we could not eliminate the overfitting completely. We have also tried adding noise to input data (Denoising AE), adding regularization (Sparse AE) to encoding layers. But still our hand made features performed better than AE created features.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Welcome to our site. Could you explain how your post answers the question in this thread? $\endgroup$
    – whuber ♦
    Feb 16, 2019 at 19:51
  • 3
    $\begingroup$ It explains where to add dropout layer in a model. Also some additional options instead of dropout. $\endgroup$
    – Jemshit
    Feb 16, 2019 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.