3
$\begingroup$

Suppose I have a neural network with several layers and on one layer I have "bad" activation function with respect to its differential properties. This function is a $max$, e.g.: $$ f(x) = max(g(x), h(x)) $$ for some $g$ and $h$.

I heard that there are some algorithms that allow us to use backpropagation even in this case, but unfortunately I can't find any of them. I looked in several books:

  • Neural Networks: A Comprehensive Foundation by Haykin
  • Deep Learning (Adaptive Computation and Machine Learning series) by Goodfellow et al.

But these books doesn't mention this topic at all.

Moreover, I heard that these methods somehow use Gumbel distribution (but I'm not sure if it is true). That's all information I have for now.

Where can I read about these algorithms of computing gradient in such cases? Unfortunately Google didn't help me with this task.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ subgradients ?? $\endgroup$
    – seanv507
    Commented Jan 29, 2021 at 21:10
  • $\begingroup$ @seanv507 it seems to be the thing I'm looking for, but are there other methods that provide such functionality? $\endgroup$
    – Georgii Firsov
    Commented Jan 29, 2021 at 22:02

1 Answer 1

Reset to default
2
$\begingroup$

$$ f' (x) = \left\{\begin{array}{ll} g' (x) & g (x) > h (x)\\ h' (x) & h (x) > g (x)\\ \text{Undefined} & g (x) = h (x), g' (x) \neq h' (x) \end{array}\right. $$

Automatic differentiation packages should be able to handle this just fine, except for the problematic $g(x)=h(x)$ case which can have issues.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.