I am training a "mini 3D Alex net" (2 convolutions + 1 max pooling + 1 dense layers) and a standard 3D Alex net on some simple synthetic data I've generated that just consists of cubes and squares (I have more complex data but I want to make sure it works on this simple test first). The mini Alex net converges relatively quickly (around 500 iterations for batch size of 30). However, the full Alex next has not converged yet and the loss functions seems to be bouncing around a small range. This may just be because the full Alex net is so much larger (especially in the 3D) that it's taking longer for all the weights to learn. But, in the case that the full alexnet is getting stuck in the local minimum, is there a reason why large nets are more likely to get stuck in local minimum?

  • $\begingroup$ As explained in this thread, computational complexity is proportional to the size of the network. Isn't it reasonable that, given more local minima, a stochastic algorithm will have a harder time finding a near-optimal solution? $\endgroup$ – Digio Jul 19 '17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.