I am using autoencoders to compress data and I see all examples on the internet using ReLU activation with image datasets. However, I am planning to use a dataset that has negative values and I was just thinking about how ReLU outputs zero if any Z values are negative. I was thinking that RELU would not be a good activation function to get my latent encoded data because of this. Would using an activation function such as sigmoid or tanh as the activation for my encoded vector be better so I always get a value?


Short answer: ReLU is not necessarily a bad choice. Try and see which works better for your particular data.

Your reasoning that ReLU will clip a portion of the data is missing one important part: before applying the activation function, the data vector is multiplied by a weight matrix. This matrix may contain negative values, so your negative inputs may yield positive values before the ReLU is applied, and conversely, positive inputs may yield negative values.

The bottom line is: you are not applying the activation function on the input values directly. Try and see whichever activation function gives you better results.

| cite | improve this answer | |
  • 1
    $\begingroup$ Ah yea that's right. Thanks for pointing that out. $\endgroup$ – zipline86 Sep 26 '18 at 12:10
  • $\begingroup$ With above reasoning, what should be the activation layer of the final output layer? Should that be not 'Linear' ? $\endgroup$ – user3698581 Feb 4 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.