2
$\begingroup$

This is not a duplicate of the Activation functions for autoencoder performing regression because there is a comment that somebody found a linear activation function but:

  1. they never said what it was.
  2. I'm looking for a linear ouput activation function that can also output negative numbers that are less than -1

I am building an autoencoder for feature reduction and have standardized my data because I have many features with different scales. However, this has caused problems because my output activation function (tanh) only outputs between 1 and -1; while some of my inputs are outside of this range. Does anybody know of any other activation function that will output numbers greater than 1 and -1 so my loss function can work with better inputs and outputs?

I have already tried min max norm to get between 0 and 1, and using a sigmoid function. But like I said, I'm working with many features with different scales and would like to see if I can get better results with standardization.

I guess what I'm specifically looking for is a output activation function that is linear; which I've never even heard of.

Anybody?

$\endgroup$
  • $\begingroup$ I found the answer to my question. There various linear activation functions I can test out as an output activation. For example, in Keras there is the keras.activations.linear(x) as well as the keras.activations.elu(x) which is exponential linear. keras.io/activations $\endgroup$ – zipline86 Sep 28 '18 at 20:54
2
$\begingroup$

In vanilla autoencoders, i.e. autoencoders with a single hidden layer, it's common to use linear activations for both the hidden and output layers. You can do it with non-linear activations for the hidden layers, but it is often imperative to use unbounded activations for the output layer, or, alternatively, transform the input to conform to the codomain of the activation function.

$\endgroup$
  • $\begingroup$ Thanks for the advice. I never thought about trying hidden activations that were linear as well. Probably because I didn't even know they existed. If anybody is using Keras, the linear activations are listed here I found the answer to my question. There various linear activation functions I can test out as an output activation. For example, in Keras there is the keras.activations.linear(x) as well as the keras.activations.elu(x) which is exponential linear. keras.io/activations $\endgroup$ – zipline86 Sep 28 '18 at 20:57
  • $\begingroup$ @zipline86 You're welcome, and welcome to the site. If you feel I answered your question, consider accepting it, clicking the green check mark (you can do it to a single answer you feel helped you the most in each question you ask). $\endgroup$ – Firebug Sep 29 '18 at 0:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.