Neural Networks - Can I Use Any Activation for the Output Layer?

Question

I'm new to neural networks, and in almost everything I'm reading, the activation function recommended on the output layer follows a specific pattern:

• If the network does binary classification (1 output node), use sigmoid
• If the network does multiclass classification (>1 output nodes), use softmax
• If the network does regression, don't use an activation function (linear)

Which I completely understand - for example, binary classification is a probability and is never going above 1 or below 0, so of course it makes sense to use sigmoid.

My question is though, when I'm doing regression, can't I just use the activation function that best fits my range of output values instead of using linear?

For example, let's say I'm trying to predict the price of a stock - wouldn't a ReLU activation function make a lot more sense to use on the output layer over a linear activation function, since the price can never be negative?

Or for another example, let's say I normalized my output values between 1 and -1 - Wouldn't I want to use a TanH activation on the output?

Answer 1

universal approximation theorem says that using linear output (and other assumptions) you can approximate any function, however if you have a bounded output, you can for sure use bounded activation...
However, ReLU would not be a nice choice for your example, and pretty much for any example, since it have a 0 gradient when less than zero... this means that initially, if the network predicts something less than zero, the ReLU would make it 0, however when calculating the loss and taking the gradient, that gradient would be zero, so the network won't correct its output.
Instead you could use $ELU(x) + 1$ which does not have zero gradient (but it can saturate, which is another small problem)

Also, keep in mind that some activations for the final layer are born with some output distribution in mind (sigmoid for Bernoulli, softmax for categorical, linear for gaussian/Laplacian and so on), and since you assume a distribution, you can optimize it via the maximum likelihood principle, from which MSE,MAE, BCE, CCE and so on come from (for example, sigmoid output layer with MSE have a problem of saturating on the extremis )