When using neural network for regression problems, the standard approach is to use tanh for hidden layer activation and linear or tanh for output layer activation.
The tanh function returns (-1, 1) and looks like the graph in the link below.
The response variable in the problem I'm looking at has high kurtosis. So the response variable looks like the tanh graph rotated around the diagonal. i.e., for tanh, as x increases, y approaches 1 but never reach 1. My response variable is the other way around even after standardizing (i.e., (x - mean(x)) / std(x)), as x increases, y increases exponentially.
In this context, the linear function seems to provide a better fit than tanh. What would be a good candidate for the activation function?
