Neural networks can have activation functions like a tanh(x), a sigmoid function, ReLU, etc.. But can we have an activation function that is a transformation of any of these functions? For instance, can we have tanh(0.5x)? Or like 100tanh(x). Does doing this pose any advantages, or does this have no difference to the neural network and just necessary? I was wondering if this might reduce the effect of saturation. Because since large x value in magnitude reaches a y value of 1 or 0 in sigmoid (which causes saturation), then doesn't that mean that it has a lower chance of being saturated if there is a function like tanh(.1x) since you need very large x values to reach 0 or 1? Or is that not right?
Thank you for your input!