0
$\begingroup$

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so backpropagating through the network will yield either 0 or infinity? I'm confused as to why it makes any sense to still use it?

A more concrete example:

Consider a network where each node in a hidden layer $n_i$ measures whether the current training point $x$ is within distance $d_i$ from some anchor point $a_i$. This can be represented as $sign(||x-a_i||_2 - d_i)$. This is a rudimentary example of a locality preserving embedding network. Naturally, the loss function will depend on the result of this node in some way. Thereofre when I try to compute the partial derivate of the loss function with respec to $a_i$ or $d_i$, it will result in a gradient of 0 and learning will not be possible.

Thanks!

$\endgroup$
1
$\begingroup$

I haven't seen a significant neural net application anywhere using the sign function as neuron activation because of the non-differentiable nature of it as you've noted down. In the paper you cited, I couldn't see a direct connection to neural-nets however as you say, they use SGD, but also note that (in Page 3) "we approximate the non-differentiable sgn function with the sigmoid". So, they actually approximate the sign function, so its derivative.

$\endgroup$
  • $\begingroup$ Very true! Thank's for your help $\endgroup$ – rohaldb Jun 19 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.