2
$\begingroup$

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum.

Will adding more neural units solve the problem?

$\endgroup$
-1
$\begingroup$

Yes, you will definitely avoid local minima with more neurons. Adding more neurons increases the volume of your solution space exponentially, but at the same time increases the amount of equivalent solutions factorially.

$\endgroup$
  • 1
    $\begingroup$ I would expect the opposite. As an extreme counter-example, consider the case where you only have 1 neuron. Then, you can always find the global optimum. $\endgroup$ – George Jun 20 '19 at 18:19
  • 1
    $\begingroup$ What guarantee do you have that this 1-dimensional function is going to be convex? $\endgroup$ – Lugi Jun 21 '19 at 12:46
  • $\begingroup$ What I had in mind is a network that consists of a single neuron which can be optimized with the perceptron algorithm. $\endgroup$ – George Jun 21 '19 at 17:29
  • $\begingroup$ The question is about gradient descent. $\endgroup$ – Lugi Jun 22 '19 at 18:05
  • 1
    $\begingroup$ this is interesting. have you got any reference? $\endgroup$ – carlo Oct 20 '19 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.