# My loss has a non-differentiable point

I had to design a loss function max(0,x). It's not differentiable at x=0. In order to train it with gradient descent, what should I do?

1. I have learned that subgradient can be used instead, so does it need to be changed in the code, or will pytorch/tf calculate subgradient automatically?
2. Or use surrogate loss, so what kind of surrogate loss is there for my loss?
• Your $\max(0,x)$ is a rectifier or ReLU activation function. One of many alternatives is the softplus or SmoothReLU function $\log_e(\exp(x)+1)$ which for large positive $x$ is close to $x$ while for large negative $x$ is close to $0$ but always has a positive derivative; you can adjust its sharpness with a parameter $k$ by using $\frac1k\log_e(\exp(kx)+1)$ which has a value of $\frac1k$ when $x=0$ Mar 17 at 10:23
• Thank you for your advice, It helped me a lot！ Mar 21 at 6:43