My teacher proved that 2nd derivate of cross-entropy is always positive, so that the cost function of neural networks using cross entropy is convex. Is this true? I'm quite confuse about this because I've always learned that the cost function of ANN is non-convex. Can anyone confirm this? Thank you a lot! http://z0rch.com/2014/06/05/cross-entropy-cost-function
The cross entropy of an exponential family is always convex. So, for a multilayer neural network having inputs $x$, weights $w$, and output $y$, and loss function $L$
is convex. However,
is not going to be convex for the parameters of the middle layer for the reasons described by iamonaboat.
What @ngiann said, and informally, if you permute the neurons in the hidden layer and do the same permutation on the weights of the adjacent layers then the loss doesn't change.
Hence if there is a non-zero global minima as a function of weights, then it can't be unique since the permutation of weights gives another global minimum. Hence the function is not convex.
The matrix of all second partial derivatives (the Hessian) is neither positive semidefinite, nor negative semidefinite. Since the second derivative is a matrix, it's possible that it's neither one or the other.
You are right in suspecting that the ANN optimisation problem of the cross-entropy problem will be non-convex. Note: we are talking about a neural network with non-linear activation function at the hidden layer. Also, non-linearity has the potential of introducing local minima in the optimization of the objective function. If you don't use a non-linear activation function then your ANN is implementing a linear function and the problem will become convex.
So the reason why the optimisation of the cross-entropy of a ANN is non-convex is because of the underlying parametrisation of the ANN. If you use a linear neural network, you can make it convex (it will essentially look like logistic regression which is a convex problem).