Suppose I have a neural network with several layers and on one layer I have "bad" activation function with respect to its differential properties. This function is a $max$, e.g.: $$ f(x) = max(g(x), h(x)) $$ for some $g$ and $h$.
I heard that there are some algorithms that allow us to use backpropagation even in this case, but unfortunately I can't find any of them. I looked in several books:
- Neural Networks: A Comprehensive Foundation by Haykin
- Deep Learning (Adaptive Computation and Machine Learning series) by Goodfellow et al.
But these books doesn't mention this topic at all.
Moreover, I heard that these methods somehow use Gumbel distribution (but I'm not sure if it is true). That's all information I have for now.
Where can I read about these algorithms of computing gradient in such cases? Unfortunately Google didn't help me with this task.