1
$\begingroup$

Are there activation functions for the output layer of a neural network whose range is $\mathbb{R}$, so that I can perform true function approximation (and not be limited to a arbitrary bounds on the output)?

I've considered using $\tanh^{-1}(2x -1)$ on the output of a sigmoid neuron since $\tanh^{-1}(x)$ has a domain of (-1, 1) and a range of $\mathbb{R}$. The problem is that $\tanh^{-1}(2\sigma(x) -1)$ is equivalent to $x/2$ (see this, so I'm not actually getting a truly unbounded function- the output of this kind of neuron is limited to [half the sum of all negative weights, half the sum of all positive weights].

Are there different combinations of output functions that can give me a truly unbounded output?

$\endgroup$
  • $\begingroup$ How do propose to train such a model, given that your training data is finite? I mean, are you doing this as a theoretical exercise on a known function? $\endgroup$ – P.Windridge Oct 29 '17 at 14:20
1
$\begingroup$

There's identity activation function. It'll simply output your $a^{[l]}=z^{[l]}$, where $z^{[l]}=\beta+w \cdot a^{[l-1]}$

With this one you can have a single layer NN that works like an ordinary least squares model with this linear activation.

There's a bunch of other unbounded functions such as bent identity and ReLU. The latter has a floor but not a cap. There's leaky ReLU which is unbounded. Follow my link to see a few function used by researchers (not so much in practice indeed)

$\endgroup$
0
$\begingroup$

You can simply carry forward the sum of the connection weights and the partial derivatives of the error w.r.t the target as the result of an activation function, and it will be unbounded. This is actually using what is called a "linear" activation function, vs. use of a logistic, tanh, exp, RBF activation function on the input-side and the output-sides. Recall, if you expand the range of the activation output in any way, e.g. via a multiplicative or power process, the range and standard deviation of the activation function outputs will increase, and lengthen the learning process.

There are some basic rules for ANNs. First, correlation between input features will cause the ANN to waste time learning the correlation, and this is why PCA and decorrelation is so important before clamping inputs to the input nodes. ANNs also like the input features to preferably be in the range [-1,1] and not even standard normally distributed using Z-scores, or even normalization or percentiles. (Percentiles will work though). You can do anything you want at the activation functions, however for a classification problem, you almost have to use the softmax function on the output side. If you are using an ANN for function approximation (continuously-scaled output value(s)), then I like to start with a linear function and then ramp up to logistic, tanh if linear doesn't reduce error at a greater rate.

$\endgroup$
  • $\begingroup$ Normalizing the inputs are OK, as long as the outputs can be extrapolated in the same way. I'm not trying to do classification, but rather function approximation (which is why I want an unbounded output). $\endgroup$ – PixelArtDragon Mar 9 '17 at 23:17
  • $\begingroup$ I'm not looking for a way to have an arbitrarily large bound on the output, I'm looking for the output to have a range of $\mathbb{R}$. It sounds like with a linear activation function, the range can only be as large as the input weights. That doesn't satisfy what I'm trying to do. $\endgroup$ – PixelArtDragon Mar 9 '17 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.