0
$\begingroup$

I had to design a loss function max(0,x). It's not differentiable at x=0. In order to train it with gradient descent, what should I do?

  1. I have learned that subgradient can be used instead, so does it need to be changed in the code, or will pytorch/tf calculate subgradient automatically?
  2. Or use surrogate loss, so what kind of surrogate loss is there for my loss?
Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Your $\max(0,x)$ is a rectifier or ReLU activation function. One of many alternatives is the softplus or SmoothReLU function $\log_e(\exp(x)+1)$ which for large positive $x$ is close to $x$ while for large negative $x$ is close to $0$ but always has a positive derivative; you can adjust its sharpness with a parameter $k$ by using $\frac1k\log_e(\exp(kx)+1)$ which has a value of $\frac1k$ when $x=0$ $\endgroup$
    – Henry
    Mar 17 at 10:23
  • $\begingroup$ Thank you for your advice, It helped me a lot！ $\endgroup$
    – firstforst
    Mar 21 at 6:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy