Suppose I have a neural network with several layers and on one layer I have "bad" activation function with respect to its differential properties. This function is a $max$, e.g.: $$ f(x) = max(g(x), h(x)) $$ for some $g$ and $h$.

I heard that there are some algorithms that allow us to use backpropagation even in this case, but unfortunately I can't find any of them. I looked in several books:

  • Neural Networks: A Comprehensive Foundation by Haykin
  • Deep Learning (Adaptive Computation and Machine Learning series) by Goodfellow et al.

But these books doesn't mention this topic at all.

Moreover, I heard that these methods somehow use Gumbel distribution (but I'm not sure if it is true). That's all information I have for now.

Where can I read about these algorithms of computing gradient in such cases? Unfortunately Google didn't help me with this task.

  • 1
    $\begingroup$ subgradients ?? $\endgroup$ – seanv507 Jan 29 at 21:10
  • $\begingroup$ @seanv507 it seems to be the thing I'm looking for, but are there other methods that provide such functionality? $\endgroup$ – Georgy Firsov Jan 29 at 22:02

$$ f' (x) = \left\{\begin{array}{ll} g' (x) & g (x) > h (x)\\ h' (x) & h (x) > g (x)\\ \text{Undefined} & g (x) = h (x), g' (x) \neq h' (x) \end{array}\right. $$

Automatic differentiation packages should be able to handle this just fine, except for the problematic $g(x)=h(x)$ case which can have issues.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.