Neural networks: how can convex optimization produce different weights each time? I am training a multilayer perceptron with a logistic activation function by backpropagation. The weights are not unique - each time I redo the fit, I get a different set of weights. However the optimization function (sum squared error in my case) is convex, which means there is one global minima.  So I suppose this minima has several equivalent sets of weights that map to it.  So how exactly do I describe this minima and what are its properties?  How do I What is the name of this phenomenon and where can I learn more about it?
 A: The loss function is not convex with respect to all of the weights.
It seems that you are getting confused because the loss function is quadratic, so you are suggesting that makes it convex.  But it is not convex with respect to all of the weights in your model.
The loss is $L = (f(w) - y)^2$ where $y$ are your targets, $f$ is your model, and $w$ are its weights.   Without the nonlinear activation functions, $f$ is a linear function, and the loss is convex.  With multiple layers having nonlinear activation functions, the loss is no longer necessarily convex.  The Hessian of $L$ will involve the Hessian of $f$, which need not be positive semidefinite.
You could reduce the loss by both increasing (or decreasing) a weight $w$, which causes a feature to light up more (or less) strongly — if you simultaneously were to change that feature's effect on the outputs (to be more or less like the targets).  This situation of being able to reduce the loss by moving in two opposite directions is like standing on the cusp between two valleys.  Move one way and you fall, move the other and you still fall.  This is not convex.
Another way to see that it's not convex is to notice that for any global minimum, there is another global minimum whose weights are a permutation of the first whereby the hidden nodes have merely been interchanged.
