Smoothness of a neural network (specifically second-order)

Question

If we use ReLU activations, then the function which our neural network represents is piecewise linear. It is not smooth and the first derivative doesn't exist everywhere.

However, if we use sigmoid or tanh activations, then the function represented by the neural network is smooth. The first derivative exists everywhere and is continuous.

I am interested to know whether we can say anything about the second derivative of a neural network with sigmoid/tanh activations. Are there any guarantees we can make about whether or not it exists everywhere? Or is this something which doesn't have a simple answer?

Answer 1

If we are talking about the usual fully connected / convolution layers combined with activations (i.e. not a complicated deep learning model with all sorts of layers), yes, they are second-order differentiable given that the activations are second-order differentiable. Or, even, we can say that, the overall function is n-th order differentiable if activations are; because the rest of the operations are just matrix multiplication.

If you think about the partial derivative wrt some variable in the network, it's just scalar multiplication, addition, activation; then the same thing again. For a two layer, it looks like

$$g(x)=f(a_2f(a_1x+b_1)+b_2)$$

which can be easily differentiated as long as $f$ is differentiable.