I am training a multilayer perceptron with a logistic activation function by backpropagation. The weights are not unique - each time I redo the fit, I get a different set of weights. However the optimization function (sum squared error in my case) is convex, which means there is one global minima. So I suppose this minima has several equivalent sets of weights that map to it. So how exactly do I describe this minima and what are its properties? How do I What is the name of this phenomenon and where can I learn more about it?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ So you are saying if the activation function and loss function are both convex then you have unique weights? Do you know if this is a necessary and/or sufficient condition? $\endgroup$– Count ZeroCommented Apr 11, 2015 at 10:12
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
The loss function is not convex with respect to all of the weights.
It seems that you are getting confused because the loss function is quadratic, so you are suggesting that makes it convex. But it is not convex with respect to all of the weights in your model.
The loss is $L = (f(w) - y)^2$ where $y$ are your targets, $f$ is your model, and $w$ are its weights. Without the nonlinear activation functions, $f$ is a linear function, and the loss is convex. With multiple layers having nonlinear activation functions, the loss is no longer necessarily convex. The Hessian of $L$ will involve the Hessian of $f$, which need not be positive semidefinite.
You could reduce the loss by both increasing (or decreasing) a weight $w$, which causes a feature to light up more (or less) strongly — if you simultaneously were to change that feature's effect on the outputs (to be more or less like the targets). This situation of being able to reduce the loss by moving in two opposite directions is like standing on the cusp between two valleys. Move one way and you fall, move the other and you still fall. This is not convex.
Another way to see that it's not convex is to notice that for any global minimum, there is another global minimum whose weights are a permutation of the first whereby the hidden nodes have merely been interchanged.
-
$\begingroup$ This is a bit short. There are more aspects to the question. Maybe you can expand your answer and address those too. $\endgroup$– AndyCommented Apr 11, 2015 at 9:10
-
$\begingroup$ @Andy: Of course it answers the question. He argued that the loss function is convex so there should be a unique solution. The mistake is the assumption. The loss function is not convex with respect to all parameters. $\endgroup$– Neil GCommented Apr 11, 2015 at 9:12
-
$\begingroup$ The previous comment was generated by the system, I changed it. Due to its brevity your post was automatically flagged as low quality. $\endgroup$– AndyCommented Apr 11, 2015 at 9:13
-
$\begingroup$ @Andy: I have no idea what to add. I answered the question as directly as possible. If it still doesn't make sense to osazuwa, I am happy to clarify things. $\endgroup$– Neil GCommented Apr 11, 2015 at 9:31
-
$\begingroup$ Hi @Andy. Thank you. Your answer makes intuitive sense. Tell me if this makes sense. Suppose Z = wY and Y = aU + bV. Then Z = waU + wbV. If your optimization function was (Z - waU - wbV)^2 then you could get unique solutions for w * a and w * b, but not for w, a, b separately. Is that right? So if I am right and the vector of products <wa, wb> is unique, is there something I can say this vector? $\endgroup$ Commented Apr 11, 2015 at 10:06