Activation function for output layer for unbounded values

Question

Are there activation functions for the output layer of a neural network whose range is $\mathbb{R}$, so that I can perform true function approximation (and not be limited to a arbitrary bounds on the output)?

I've considered using $\tanh^{-1}(2x -1)$ on the output of a sigmoid neuron since $\tanh^{-1}(x)$ has a domain of (-1, 1) and a range of $\mathbb{R}$. The problem is that $\tanh^{-1}(2\sigma(x) -1)$ is equivalent to $x/2$ (see this, so I'm not actually getting a truly unbounded function- the output of this kind of neuron is limited to [half the sum of all negative weights, half the sum of all positive weights].

Are there different combinations of output functions that can give me a truly unbounded output?

Answer 1

There's identity activation function. It'll simply output your $a^{[l]}=z^{[l]}$, where $z^{[l]}=\beta+w \cdot a^{[l-1]}$

With this one you can have a single layer NN that works like an ordinary least squares model with this linear activation.

There's a bunch of other unbounded functions such as bent identity and ReLU. The latter has a floor but not a cap. There's leaky ReLU which is unbounded. Follow my link to see a few function used by researchers (not so much in practice indeed)

Answer 2

You can simply carry forward the sum of the connection weights and the partial derivatives of the error w.r.t the target as the result of an activation function, and it will be unbounded. This is actually using what is called a "linear" activation function, vs. use of a logistic, tanh, exp, RBF activation function on the input-side and the output-sides. Recall, if you expand the range of the activation output in any way, e.g. via a multiplicative or power process, the range and standard deviation of the activation function outputs will increase, and lengthen the learning process.

There are some basic rules for ANNs. First, correlation between input features will cause the ANN to waste time learning the correlation, and this is why PCA and decorrelation is so important before clamping inputs to the input nodes. ANNs also like the input features to preferably be in the range [-1,1] and not even standard normally distributed using Z-scores, or even normalization or percentiles. (Percentiles will work though). You can do anything you want at the activation functions, however for a classification problem, you almost have to use the softmax function on the output side. If you are using an ANN for function approximation (continuously-scaled output value(s)), then I like to start with a linear function and then ramp up to logistic, tanh if linear doesn't reduce error at a greater rate.