1
$\begingroup$

Neural networks can have activation functions like a tanh(x), a sigmoid function, ReLU, etc.. But can we have an activation function that is a transformation of any of these functions? For instance, can we have tanh(0.5x)? Or like 100tanh(x). Does doing this pose any advantages, or does this have no difference to the neural network and just necessary? I was wondering if this might reduce the effect of saturation. Because since large x value in magnitude reaches a y value of 1 or 0 in sigmoid (which causes saturation), then doesn't that mean that it has a lower chance of being saturated if there is a function like tanh(.1x) since you need very large x values to reach 0 or 1? Or is that not right?

Thank you for your input!

$\endgroup$

1 Answer 1

2
$\begingroup$

Scaling input or output of activation function doesn't exactly solve the problems you mentioned, because this scaling is absorbed in model's weights. If you multiply activation input, it gets absorbed in the input weights. If you multiply the output, it gets absorbed in the next layer's weights. This basically changes the scale of weights.

Carefully adjusting weight scales is really useful though. But it's usually only done at initialization - you might want to read on He or Xavier initialization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.