Suppose I have a neural network with several layers and on one layer I have "bad" activation function with respect to its differential properties. This function is a $max$, e.g.: $$ f(x) = max(g(x), h(x)) $$ for some $g$ and $h$.

I heard that there are some algorithms that allow us to use backpropagation even in this case, but unfortunately I can't find any of them. I looked in several books:

  • Neural Networks: A Comprehensive Foundation by Haykin
  • Deep Learning (Adaptive Computation and Machine Learning series) by Goodfellow et al.

But these books doesn't mention this topic at all.

Moreover, I heard that these methods somehow use Gumbel distribution (but I'm not sure if it is true). That's all information I have for now.

Where can I read about these algorithms of computing gradient in such cases? Unfortunately Google didn't help me with this task.

  • 1
    $\begingroup$ subgradients ?? $\endgroup$
    – seanv507
    Jan 29, 2021 at 21:10
  • $\begingroup$ @seanv507 it seems to be the thing I'm looking for, but are there other methods that provide such functionality? $\endgroup$ Jan 29, 2021 at 22:02

1 Answer 1


$$ f' (x) = \left\{\begin{array}{ll} g' (x) & g (x) > h (x)\\ h' (x) & h (x) > g (x)\\ \text{Undefined} & g (x) = h (x), g' (x) \neq h' (x) \end{array}\right. $$

Automatic differentiation packages should be able to handle this just fine, except for the problematic $g(x)=h(x)$ case which can have issues.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.