0
$\begingroup$

I hear a lot about local minima for neural networks. I understand the theory behind it - but if my neural network finds weights in a local minimum, is that a bad thing?

I understand that finding global minima (in Neural Networks) is usually a bad thing as well, since global minima usually overfits. However I am still a bit confused.

Do convergence to local minima give us bad solutions? Lastly, does convergence to local minima overfit to our training data?

$\endgroup$

2 Answers 2

1
$\begingroup$
  1. You can consider local minima L bad if a) your model does not overfit on L and b) there's some other minima L' which has significantly lower CV error rate than L.
  2. Global minima in NN is not usually a bad thing. It is bad only if your model overfits, but you can use always proper regularization and stop early.
  3. Overfitting has nothing to do with convergence, a model can overfit long before convergence.
$\endgroup$
0
$\begingroup$

Local Minima are simply Attractors for the Training Dynamic, it’s the Associated Network Performance that may be good or bad.

Moreover I think that typically the same performance level can be associated to multiple local minima as a result of the inference function invariance to some neurons permutation: e.g. consider a MLP with Fully Connected Topology, each Neuron Input is a Weighted Sum of all the Previous Layer Neurons Outputs so if you permute the Previous Layer Neurons and the corresponding Input Weights at each Next Layer Neuron you should land in a different point in the Parameters Space with the same Final Performance

Edit

I found that also Bishop in his famous "Pattern Recognition and Machine Learning" (2006) in Chap 5.1.1 "Weight Space Symmetries" refers to this phenomenon

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.