# Gradient calculation with backpropagation through $max$ function

Suppose I have a neural network with several layers and on one layer I have "bad" activation function with respect to its differential properties. This function is a $$max$$, e.g.: $$f(x) = max(g(x), h(x))$$ for some $$g$$ and $$h$$.

I heard that there are some algorithms that allow us to use backpropagation even in this case, but unfortunately I can't find any of them. I looked in several books:

• Neural Networks: A Comprehensive Foundation by Haykin
• Deep Learning (Adaptive Computation and Machine Learning series) by Goodfellow et al.

But these books doesn't mention this topic at all.

Moreover, I heard that these methods somehow use Gumbel distribution (but I'm not sure if it is true). That's all information I have for now.

Where can I read about these algorithms of computing gradient in such cases? Unfortunately Google didn't help me with this task.

• subgradients ?? – seanv507 Jan 29 at 21:10
• @seanv507 it seems to be the thing I'm looking for, but are there other methods that provide such functionality? – Georgy Firsov Jan 29 at 22:02

## 1 Answer

$$f' (x) = \left\{\begin{array}{ll} g' (x) & g (x) > h (x)\\ h' (x) & h (x) > g (x)\\ \text{Undefined} & g (x) = h (x), g' (x) \neq h' (x) \end{array}\right.$$

Automatic differentiation packages should be able to handle this just fine, except for the problematic $$g(x)=h(x)$$ case which can have issues.