My teacher proved that 2nd derivate of cross-entropy is always positive, so that the cost function of neural networks using cross entropy is convex. Is this true? I'm quite confuse about this because I've always learned that the cost function of ANN is non-convex. Can anyone confirm this? Thank you a lot! http://z0rch.com/2014/06/05/cross-entropy-cost-function
$\begingroup$
$\endgroup$
1
-
7$\begingroup$ Link is broken? $\endgroup$– call-in-coJul 27, 2015 at 16:14
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
The cross entropy of an exponential family is always convex. So, for a multilayer neural network having inputs $x$, weights $w$, and output $y$, and loss function $L$
$$\nabla^2_y L$$
is convex. However,
$$\nabla^2_w L$$
is not going to be convex for the parameters of the middle layer for the reasons described by iamonaboat.
$\begingroup$
$\endgroup$
1
What @ngiann said, and informally, if you permute the neurons in the hidden layer and do the same permutation on the weights of the adjacent layers then the loss doesn't change.
Hence if there is a non-zero global minima as a function of weights, then it can't be unique since the permutation of weights gives another global minimum. Hence the function is not convex.
The matrix of all second partial derivatives (the Hessian) is neither positive semidefinite, nor negative semidefinite. Since the second derivative is a matrix, it's possible that it's neither one or the other.
-
2$\begingroup$ If you want to be be pedantic rather than informal, the usual definition of function convexity does not require a unique global minimum, so non-uniqueness of the minima doesn't imply non-convexity. However, permuting the weights in this way doesn't change the actual output of your network either, so even without convexity, you could still have the property that your training function converges to the same ~function every time. I bet there is a way to break this symmetry to order the weights without screwing up other properties of the loss function. $\endgroup$ Sep 23, 2016 at 9:59
$\begingroup$
$\endgroup$
You are right in suspecting that the ANN optimisation problem of the cross-entropy problem will be non-convex. Note: we are talking about a neural network with non-linear activation function at the hidden layer. Also, non-linearity has the potential of introducing local minima in the optimization of the objective function. If you don't use a non-linear activation function then your ANN is implementing a linear function and the problem will become convex.
So the reason why the optimisation of the cross-entropy of a ANN is non-convex is because of the underlying parametrisation of the ANN. If you use a linear neural network, you can make it convex (it will essentially look like logistic regression which is a convex problem).
