0
$\begingroup$

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \begin{cases} 1, &x \geq \gamma\\ 0, &x < \gamma \end{cases} $$

$\endgroup$
4
$\begingroup$

The function you wrote is completely flat, except at $x=\gamma$ where the step occurs. So, its derivative w.r.t. $x$ is zero everywhere, except at $x=\gamma$, where the derivative doesn't exist. This means gradients (whenever they exist) will always be zero, and gradient-based learning rules (e.g. backprop) won't work. If you want learning to happen, you'd have to use a non-flat activation function or a learning rule that isn't based on gradients.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.