I was training a singled layered (shallow) neural network as in:

$$ f(x) = \sum^K_{k} c_k\theta(W_k x+b_k)$$

for regression (using squared error loss) or function approximation.

I was wondering, is there a preference on what type of activation one should use? For example, sigmoid, ReLu, tanh, etc? I am aware that ReLu are usually the preferred method when using multiple layers to combat the vanishing gradient problem, however, it doesn't seem the argument there applies to shallow neural networks. Is there a particular advantage of using any activation function compared to another one in this special case?

The only things I thought (intuitively) is that sigmoids or tanh seem more non-linear that ReLu's and since its singled layered, I'd probably prefer a more "non-linear model" though these are just guesses. If there is better research for this I'd love to know!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.