I am training a multilayer perceptron with a logistic activation function by backpropagation. The weights are not unique - each time I redo the fit, I get a different set of weights. However the optimization function (sum squared error in my case) is convex, which means there is one global minima. So I suppose this minima has several equivalent sets of weights that map to it. So how exactly do I describe this minima and what are its properties? How do I What is the name of this phenomenon and where can I learn more about it?

  • $\begingroup$ So you are saying if the activation function and loss function are both convex then you have unique weights? Do you know if this is a necessary and/or sufficient condition? $\endgroup$ – Count Zero Apr 11 '15 at 10:12

The loss function is not convex with respect to all of the weights.

It seems that you are getting confused because the loss function is quadratic, so you are suggesting that makes it convex. But it is not convex with respect to all of the weights in your model.

The loss is $L = (f(w) - y)^2$ where $y$ are your targets, $f$ is your model, and $w$ are its weights. Without the nonlinear activation functions, $f$ is a linear function, and the loss is convex. With multiple layers having nonlinear activation functions, the loss is no longer necessarily convex. The Hessian of $L$ will involve the Hessian of $f$, which need not be positive semidefinite.

You could reduce the loss by both increasing (or decreasing) a weight $w$, which causes a feature to light up more (or less) strongly — if you simultaneously were to change that feature's effect on the outputs (to be more or less like the targets). This situation of being able to reduce the loss by moving in two opposite directions is like standing on the cusp between two valleys. Move one way and you fall, move the other and you still fall. This is not convex.

Another way to see that it's not convex is to notice that for any global minimum, there is another global minimum whose weights are a permutation of the first whereby the hidden nodes have merely been interchanged.

  • $\begingroup$ This is a bit short. There are more aspects to the question. Maybe you can expand your answer and address those too. $\endgroup$ – Andy Apr 11 '15 at 9:10
  • $\begingroup$ @Andy: Of course it answers the question. He argued that the loss function is convex so there should be a unique solution. The mistake is the assumption. The loss function is not convex with respect to all parameters. $\endgroup$ – Neil G Apr 11 '15 at 9:12
  • $\begingroup$ The previous comment was generated by the system, I changed it. Due to its brevity your post was automatically flagged as low quality. $\endgroup$ – Andy Apr 11 '15 at 9:13
  • $\begingroup$ @Andy: I have no idea what to add. I answered the question as directly as possible. If it still doesn't make sense to osazuwa, I am happy to clarify things. $\endgroup$ – Neil G Apr 11 '15 at 9:31
  • $\begingroup$ Hi @Andy. Thank you. Your answer makes intuitive sense. Tell me if this makes sense. Suppose Z = wY and Y = aU + bV. Then Z = waU + wbV. If your optimization function was (Z - waU - wbV)^2 then you could get unique solutions for w * a and w * b, but not for w, a, b separately. Is that right? So if I am right and the vector of products <wa, wb> is unique, is there something I can say this vector? $\endgroup$ – Count Zero Apr 11 '15 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.