# When doing regression with a singled layered Neural Network, what activation function is the best one to choose?

I was training a singled layered (shallow) neural network as in:

$$f(x) = \sum^K_{k} c_k\theta(W_k x+b_k)$$

for regression (using squared error loss) or function approximation.

I was wondering, is there a preference on what type of activation one should use? For example, sigmoid, ReLu, tanh, etc? I am aware that ReLu are usually the preferred method when using multiple layers to combat the vanishing gradient problem, however, it doesn't seem the argument there applies to shallow neural networks. Is there a particular advantage of using any activation function compared to another one in this special case?

The only things I thought (intuitively) is that sigmoids or tanh seem more non-linear that ReLu's and since its singled layered, I'd probably prefer a more "non-linear model" though these are just guesses. If there is better research for this I'd love to know!