I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so backpropagating through the network will yield either 0 or infinity? I'm confused as to why it makes any sense to still use it?
A more concrete example:
Consider a network where each node in a hidden layer $n_i$ measures whether the current training point $x$ is within distance $d_i$ from some anchor point $a_i$. This can be represented as $sign(||x-a_i||_2 - d_i)$. This is a rudimentary example of a locality preserving embedding network. Naturally, the loss function will depend on the result of this node in some way. Thereofre when I try to compute the partial derivate of the loss function with respec to $a_i$ or $d_i$, it will result in a gradient of 0 and learning will not be possible.